What Maths subjects should I study to understand Quantum Mechanics and General Relativity?
As far as I know, in order to fully understand GR, you might want to take a Tensor Analysis Class, along with a differential geometry class. For QM, i think you basically the mathematics required when you are learning it.
For GR you need tensor analysis, differential geometry, and for a sufficiently sophisticated course, topology.
For QM you need differential equations, linear algebra, and group theory.
Hilbert spaces wouldn't go amiss and if you want to look to the future then category theory too.
Where does one use topology in general relativity?
All over de place.
Check out this link:
Do a search for "topology" and see all the fun places it takes you!
Tensor geometry, diff topology, and functional analysis are very broad subjects. A lot of what you study in a strict maths class you won't really use in physics. Generally the mathematics you need will be taught in the physics class.
Of course, if you have an interest in doing further work/research in mathematical physics then doing the extra maths is very helpful because you may find interesting applications.
Separate names with a comma.