It really depends on how much you are interested in the (rigorous) mathematical foundations of QM. If you are only interested in learning the basics of QM and how to use it, then a solid knowledge of finite dimensional linear algebra supplemented with some (very basic) material from functional analysis is enough (of course, assuming you know calculus, differential equations, and all the usual stuff in the physicist's toolkit)
If you are really interested in the mathematical foundations, then you need some good amount of stuff: basics of general topology, basics on measure theory (including its application to probability theory; the quantum logic formulation of QM is a generalization of this), Banach spaces and algebras (including spaces of operators), Hilbert spaces (the basic stuff like Riesz's theorem and bases, but also the general theory behind bounded operators and densely-defined unbounded operators); spectral theory (the spectral theorem for unbounded self-adjoint operators and the theory behind it); Lie groups and harmonic analysis (including the imprimitivity theorem; most of the foundational issues related to the study of basic quantum systems, like localizable and covariant systems, can be reduced to the study and application of this theorem, i.e., the classification of representations of different Lie groups).
Most mathematicians are already familiar with all this material, so I don't think you will find anything new (I say this because, if I remember well, you are a mathematician)
