You're first talking about viewing a field as a mathematical definition, which is correct in the classical sense. More general, some field configuration has as an input coordinates of space-time and as an output it can have numbers (scalar fields), vectors (vector fields), tensors, spinors, etc. So far so good.
But next you turn to question of energy, and you somehow step away from this mathematical treatment.
The fact is, is that the energy associated with some field is also just a mathematical relation. For instance, the energy can be defined in terms of the derivative of the field, or the value of the field squared (yes, the energy is then also coordinate dependent! So it's actually an energy density).
The problem lies in viewing the concept of energy on some special, non-mathematical footing - which is wrong. Energy is itself a descriptive quantity, which we can use to describe the fields and their dynamics. What makes energy so special is that, to a large extent, it determines the dynamics of a system.