You can get very far on a solid understanding of a small range of topics in linear algebra. (linear independence, bases,
the relationship between linear operators and matrices, eigenvectors, inner products, orthonormal bases). You need very little from calculus. For example, you need to understand what an integral
is, but you don't have to know how to integrate weird combinations of elementary functions. I would say that you don't need anything from differential equations. The QM book will tell you what you need to know.
There was a "The mathematical methods of physics" class given at the physics department of my university, but it was useless for someone who just wants to understand QM (and even more useless for someone who wants to understand general relativity). I'm not sure who it was supposed to be useful for. A book like Arfken covers a lot of stuff that you really don't need, and isn't a very good place to learn the things you
do need. You're better off studying a good linear algebra book. (My favorite is Axler).
I should also say that there's an enormous difference between "the mathematics you need to understand QM" and "the mathematics of QM". I would say that you need the latter to
really understand QM, but most physicists know very little about it. To learn the mathematics of QM, you need at least one, probably two, courses on functional analysis. (One of them should include operator algebras). Unfortunately you can't even begin to read most books on functional analysis if you don't know general topology (a.k.a. point set topology), and you would find a topology book really hard if you haven't taken an advanced course in real analysis first. (Something like Rudin's "Principles of mathematical analysis").
I've been told that the functional analyis book by Kreyszig makes things much easier for its readers, so maybe it doesn't have to be as hard as it was for me to get started with these things.
So you would have to take about 4 additional math courses that physics students don't normally take to understand the mathematics of QM. To understand the mathematics of quantum field theories, you also need one, probably two, courses on differential geometry (one of them should include fiber bundles), and a course on representation theory. So that's 3 more. Of course, you don't need any of those courses to be able to take a course in quantum field theory. There's always a big difference between a branch of physics and the mathematics associated with it.