Silviu said:
I assume this is not used just to write old stuff in a nicer way, but it also gives new insight
Well, it gives you a better mathematical insight, hence you can apply mathematical theorems possibly more directly. For example it becomes much clearer, when and why the scalar and vector potentials exist. (this is connected to the concept of de Rham cohomology. The natural reference for this is:
Bott, Tu - Differential Forms)
Also it makes the transition to GR and gauge theories easier. The formulation of ED in terms of differential forms is useful because you can in fact directly generalize it to the Maxwell-part of Einstein-Maxwell-Theory and also it is nice to relate it more directly to gauge theory, where the gauge field ##A## (=vector potential) is naturally written as a 1-form. Furthermore, if you can deal with it, it makes most calculations much shorter.
For a nice mathematical treatment of Einstein-Maxwell-Theory, see
Straumann - General Relativity.
Gauge theory (and also a nice treatment of ED) is incorporated in
Baez, Muniain - Gauge Fields, Knots and Gravity
Further general references:
Nakahara - Geometry, Topology and Physics
Frankel - The Geometry of Physics
Naber - Topology, Geometry and Gauge Fields I and II
Naber - The Geometry of Minkowski Space Time
In all of these you will find more advanced stuff and more basic stuff, as well as physical applications and pure mathematics.
If you are capable of reading and understanding German, I would also recommend
Knauff - Mathematische Physik-Klassische Mechanik
For Hamiltonian mechanics you could also search for books and lecture notes on symplectic geometry, e.g.
https://people.math.ethz.ch/~acannas/Papers/lsg.pdf
The symplectic approach to Hamiltonian mechanics is for example used in the Deformation Quantization approach to Quantum Mechanics, where you deform the symplectic structure on phase space in order to represent Quantum Mechanics purely algebraically in terms of classical functions.
See, e.g.
Waldmann - Recent Developments in Deformation Quantization