As for the question "what good does formalism do?", while it may not cast significant light on the physics in and of itself (but I think this is subjective), or be useful in computations, it may provide clues about how to generalize a theory, or hints toward unification.
However, often formalism goes a long way. For example, some basic things about qm, like the impossibility of duplicating states, follows quite easily from the Hilbert space formalism.
For the "basic" stuff, e.g. general relativity and nonrelativistic qm, you would be well off with elementary differential geometry and functional analysis. Classical mechanics basically studies flows on symplectic manifolds, etc.
I think this article:
http://ncatlab.org/nlab/show/physics and the ones it links to, documents most of the mathematics used in modern physics.
For the really heavy duty stuff at the end of the spectrum, e.g. axiomatic qft, you need just about everything. There is a kind of duality going on, where you have equivalent algebraic and geometric formalisms. Check out the "Related consepts" at ncatlab.org/nlab/show/AQFT