Personally, I always learn math much better when I can see what it's supposed to be doing in physics (I struggle with physics much less than math). At least in introductory physics classes, while they are calculus based, in my experience they haven't been very rigorous with the calculus - by that I mean the physics professors are trying to see if you can do physics, not calculus. You may have to do an integral or a derivative, but the difficulty in the problem is understanding how to set up the integral, or why you need the derivative - not the computation itself.