I don't know much about astrophysics books, so I will only restrict this to the undergrad courses.
The first step must be to learn single variable calculus and then multivariable calculus. But you will also need to study vectors, (at least some) three dimensional geometry, trigonometry. Perhaps other than calculus, you already have some exposure to these areas. Partial derivatives, which fall under multivariable calculus are absolutely unavoidable in college physics. But this math will also take some time to sink in, and it is only after it does sink in will you be able to appreciate the physics better.
Some good book recommendations have been made for you here, but most of these books assume grounding in the math topics mentioned above. Then, as you go along, you learn some more math. If you are in your first year in college and learning calculus through some courses, you should do that first. Resnick, Halliday and Krane (2 volumes) is, imho, much better than Resnick, Halliday, Walker (the single fat book). The latter has compromised severely on the math and leaves much to be desired (at least till the sixth edition, which is what I had referred to).
Griffiths' textbook on EM was the one we exclusively used in our second semester course in electromagnetism. In fact, I haven't come across any other book that is being so extensively used. But then we had already studied calculus in school, and the multivariable calculus we needed was taught to us in the first semester in college. As for mechanics, we used Kleppener and Kolenkow, but by the time the interesting stuff came in, we had already studied partial derivatives, so we could also refer to the book by Goldstein occasionally. There are of course many others, but Kleppener and Kolenkow seems to be pedagogically one of the best...also have a look at the book by AP French.
For Optics, I also recommend the book by Hecht. For modern physics, the two books by Arthur Beiser and Sproull are very good.
Once you have taken the introductory courses, you can study the books on EM by Jackson, mechanics by Goldstein, quantum mechanics by Griffiths (and a whole lot of other books, lots of variety here), etc.
Anyway the crucial thing to realize is that the math is very important here...you simply cannot afford to compromise your math training. There's a nice book by R. Shankar which might actually help you speed up stuff:
https://www.amazon.com/dp/0306450364/?tag=pfamazon01-20.
Finally, always keep Berkeley Physics Course volumes and Feynman's Lectures on Physics handy...they're
always helpful.