In physics you need linear algebra and calculus! I think the most difficult thing in starting to learn physics is that one has to deal with vector or even tensor calculus before one has studied these subjects in the math course.
One should not learn these topics with quantum theory, because the challenge to comprehend quantum theory shouldn't be the mathematics since the really weird thing about it is the physics. I recommend to learn vector calculus first by studying classical physics (fluid dynamics and classical electromagnetism). Most intuitive is to have hydrodynamics as a picture for the various operations (div, grad, curl and various types of integrals related with these differential operators, Stokes's, Gauss's, and Helmholtz's theorems, etc.).
A short, but the the most useful intuitive introduction into this can be found in Sommerfeld's Lectures on Theoretical Physics, Vol. II.
Also, one should not start to learn quantum theory with photons, which is even more difficult than non-relativistic quantum mechanics of massive particles. The Feynman Lectures are a good starting point, particularly because of Feynman's famous "no-nonsense approach" to the subject. However, as many textbooks, it's overemphasizing the wave-mechanics formulation. After some familiarity with this approach one should come soon to the more abstract formulation in terms of Dirac's bra-ket formalism! A good intro for that is J. J. Sakurai, Modern Quantum Mechanics.