If you are no good at algebra (and live in the US) go to your local community college and for a couple hundred bucks you can get a 16 week course that you can ask all of your questions in. IF you don't know calculus, go enroll in Calc 1, then Calc 2, then Calc 3 and then Differential equations. Then take a linear algebra class. All of these classes can be taken at a local community college for not much money considering the personal attention you will receive. Once you have done all of that (or perhaps during the last half) you are ready to take the two normal introductory courses in physics (Physics with calculus 1 and 2). Once you have done all of that, go to a university and take a mathematical methods in physics course, then mechanics, then electricity and magnetism, and finally a mathematical physics course. Then you can pretty much do anything: read, watch susskind's online lectures (the theoretical minimum), etc.
PS: This can be done and not only that, it is extremely fun. If you have discipline and a genuine concern for the math behind it all then this course of events will be mind-blowing to you in a wonderful way. If you are really disciplined, have proper guidance and do all of the ancillary stuff, then with a little more work (a few more classes) you can get a BS degree in physics and end up in a top physics graduate program. Then you'll be immersed in this stuff, which is even better.