Espen's answer is good, but I disagree about the differential equations (and the Fourier series). People always mention differential equations as a prerequisite to QM in these threads (there are many of them; you might find them useful, so I suggest you search for them). I have never understood why. You will only encounter
one differential equation in QM, and the book will tell you how to solve it. Similar comments apply to all of the classical theories, with the exception of classical electrodynamics. (If you're not going to get deep into the engineering applications of electrodynamics, you don't have to worry about those differential equations either). If you ever get to general relativity, you will have to deal with an equation that's so hard to solve that you will only be looking at the two or three simplest solutions, and the books will tell you how to find those.
So my advice is: Don't worry about differential equations. Just let the physics books teach you what you need to know about them.
I think it's reasonable for someone like you to study calculus, linear algebra, non-relativistic classical mechanics, special relativity and quantum mechanics, but it's likely that you won't be able to study general relativity, quantum field theory, and the mathematics of quantum mechanics in detail any time soon. (Most physicists never study the mathematics of QM. They just study linear algebra to get
some idea what they're doing).
Linear algebra is extremely useful in QM, and the basics are very useful in SR too, so don't underestimate its importance.A few book recommendations:
Calculus: Any introductory book will do. The one by Serge Lang looks really good to me.
Linear algebra: I really like Axler. Some people will say that there are easier introductions, and others will say that there are books with a more complete coverage of linear algebra, but I believe that this is the best book for someone who intends to study QM.
Classical mechanics: I don't know what to recommend here. I studied Kleppner & Kolenkow a long time ago. It wasn't bad, but I have a feeling there are better books.
Special relativity: Taylor & Wheeler gets the most recommendations. I haven't read it myself, but I don't doubt that it's good. The best intro I've actually read is the part about SR in Schutz's GR book.
Quantum mechanics: Griffiths is a nice introduction, and Isham is an excellent supplement to it. Ballentine is a good book for advanced students (but is probably not of any use to you right now). You should also read "QED: The strange theory of light and matter", by Richard Feynman (lectures about light for people who don't know mathematics).
General relativity (without mathematics): "Black holes and time warps: Einstein's outrageous legacy", by Kip Thorne.
General relativity (with mathematics): I'll just quote this guy:
Nabeshin said:
There are really three tiers at which you can really learn anything substantive about general relativity.
Tier I: Primarily algebra based with light calculus. This is suitable for a first or second year university student. Prior knowledge of physics should include the basic rudiments of mechanics from the F=ma point of view. The hallmark textbook is:
https://www.amazon.com/dp/020138423X/?tag=pfamazon01-20
Tier II: Calculus based with light differential geometry. This is suitable for a third or fourth year university student. Prior knowledge of physics should include an upper level E&M course and exposure to lagrangians and hamiltonians. The hallmark textbook is:
https://www.amazon.com/dp/0805386629/?tag=pfamazon01-20
Tier III: Full general relativity with differential geometry. This is usually relegated to graduate classes, but can be tackled by advanced undergraduates. Prior physics knowledge is basically an entire undergraduate physics/astrophysics education. Some representative textbooks are:
https://www.amazon.com/dp/0226870332/?tag=pfamazon01-20
https://www.amazon.com/dp/0716703440/?tag=pfamazon01-20
Differential geometry: "Introduction to smooth manifolds" and "Riemannian manifolds: an introduction to curvature", both by John M. Lee.