Electric to be said:
The Hamiltonian in classical mechanics is not always equal to the total energy of the system. I believe this is only true if there is only a potential field and no vector potential.
Electric to be said:
So my question still kind of remains. If there is a non-conservative field, say an E field with Curl. The Hamiltonian will still be defined as the operator for energy in QM, but how is energy in general defined then? (since there isn't any scalar potential energy).
Energy is defined in general as follows for the case I think you have in mind. We can split the Lagrangian ##L(q,\dot{q})=T(q,\dot{q})-U(q,\dot{q})## into a quadratic part (kinetic) and whatever remains, as
$$L=\frac{1}{2}\sum_{ij}c_{ij}\dot{q}_i\dot{q}_j-U(q,\dot{q}).$$
##U## is the generalised potential, which will contain both scalar and vector potentials. The Hamiltonian is defined in terms of the Legendre transform $$H=\sum_i \frac{\partial L}{\partial\dot{q}_i}\dot{q}_i-L=\sum_i p_i\dot{q}_i-L.$$ In terms of the above expression for ##L##, the canonical momenta read $$p_k=\sum_{ki}c_{ki}\dot{q}_i-\frac{\partial U}{\partial \dot{q}_i},$$
where we use the fact that ##c_{ij}=c_{ji}##. In the case of electromagnetism, the 2nd term on the right hand side of the above expression is ##-A_i##, i.e. the negative of the vector potential.
Inserting the canonical momenta into the above Legendre tranform, we find that $$H=\sum_{ij}c_{ij}\dot{q}_i\dot{q}_j-\sum_i\frac{\partial U}{\partial\dot{q}_i}\dot{q}_i-L=T+U-\sum_i\frac{\partial U}{\partial\dot{q}_i}\dot{q}_i$$.
This is a general expression for the total energy of a system with fixed energy (may need to be a Galilean invariant system though) when the generalised potential is velocity dependent (Is this the general expression for the energy you are looking for?). The point is that it isn't the scalar potential energy ##V## which is the fundamental quantity but the generalised scalar potential ##U##. The vector potential can be derived from ##U## through
partial differentiation, the components of ##A## are the slopes of ##U## with respect to the velocities. Any curl of an electric field etc.. will be encoded in this.
However you can still define the total energy as being "kinetic" + "potential" energies: ##H=T+V##, so long as you define $$V=U-\sum_i\frac{\partial U}{\partial\dot{q}_i}\dot{q}_i.$$
In the case of EM: $$U=-\sum_i A_i\dot{q}_i+\phi$$ and so the potential energy ##V## is ##\phi## and the Hamiltonian is $$H=\frac{(\mathbf{p}-\mathbf{A})^2}{2m}+\phi=T+V.$$
You can also see that this gives the correct expression for the Lorentz force, as follows.
The Euler-Lagrange equations $$\frac{\partial L}{\partial q_k}-\frac{d}{dt}\left(\frac{\partial L}{\partial\dot{q}_k}\right)=0,$$ splits off into a generalised force (The F in Newton's equation, here associated with the generalised potential): $$F_k=-\frac{\partial U}{\partial q_k}+\frac{d}{dt}\left(\frac{\partial U}{\partial\dot{q}_k}\right),$$
and a force of inertia (The ##ma## in Newton's equation, here absorbed in the kinetic energy function): $$-\frac{\partial T}{\partial q_k}+\frac{d}{dt}\left(\frac{\partial T}{\partial\dot{q}_k}\right)=F_k.$$ Substituting the earlier general definition of ##V## in terms of ##U## into the above expression for ##F_k##, you find that the generalised force takes the Lorentz form $$F_k=-\frac{\partial V}{\partial q_k}-\frac{\partial A_k}{\partial t}+\sum_j\dot{q}_j\left(\frac{\partial A_j}{\partial q_k}-\frac{\partial A_k}{\partial q_j}\right),$$
where ##A_i=\frac{\partial U}{\partial\dot{q}_i}## and we use the fact that the total time derivative $$\frac{dA_k}{dt}=\frac{\partial A_k}{\partial t}+\sum_i\dot{q}_i\frac{\partial A_k}{\partial q_i}.$$
Does this help at all?