I think this is a dilemma many mathematically-minded physicists have. There really are two kinds of physicists: those who don't really care about math (they just use it fairly loosely, don't care about the proofs, and memorize mathematical facts rather than deriving them), and those who really do want to understand the math deeply.
Studying physics quickly leads you into all kinds of advanced math--even sophomores in undergrad physics will learn to use spherical harmonics, Fourier transforms, legendre transformations, gaussian/fourier integrals, etc. These are all advanced topics that you'd be lucky to cover rigorously in four years as an undergrad math major.
One of my favorite examples of this kind of thing is when a physics teacher writes something like this on the board:
[tex]\int_{-\infty}^{\infty}\sum_{n=1}^\infty F_n(x)dx[/tex]
Almost invariably they'll quickly swap the summation symbol and integral symbol, hoping that nobody in the class notices... Rarely they'll mutter something like "And in case you're wondering, this series is uniformly convergent." Uniform convergence is a big deal to math majors, and typically they don't cover it until a senior-level analysis course (they'll usually dwell on uniform convergence for about a week in an analysis class).
Another example is Lie groups. Physicists use Lie groups all the time, but even after two senior-level undergrad courses in linear algebra and abstract algebra, you'd be lucky to even touch on Lie groups (they typically spend the entire semester on finite groups and the trivial infinite groups like Z, R, C, etc.).
One last example: the "dirac delta function". Mathematicians cringe at that phrase, since it's really a distribution rather than a function. The fact that physicists use it ALL the time and call it by the erroneous name just shows that physicists are more concerned with quick and dirty tools rather than a mathematically correct understanding.
The point I'm trying to make is that to
study mathematics rigorously, you take a LOT longer to get to the advanced topics. You could spend years trying to rigorously understand Fourier transforms, so it's not recommended you try to learn all the math rigorously before using it in physics.
But knowing the math definitely helps you with physics. I definitely don't want to give the impression that it's useless to learn the math rigorously. Learning contour integration, PDEs, and linear (matrix) algebra the correct mathematical way will really help any physicist.
The fact is that if you're a typical physics student, there will always be more math to learn, and the more physics you study, the more math you'll feel you need to understand. The only way you could ever avoid being in that situation is by devoting years and years to math (like getting a Ph.D. in math) and only THEN turning to physics--e.g. David Hilbert.