Well, the OP is really asking HOW MUCH TIME that s/he would spend on math, so I am very confused.
Because if one is already in college, or even, as someone suggest earlier, in the grad school already, he or she should have some basic idea of how much math is involved. So I couldn't resist to wonder why the OP is asking this question?
My answer is, it depends. Do you just want to take all the basic math classes that physics major requires? Or are you curious about all the advance, proof base (and fascinating in my opinion!) math? And then, what are you expecting to get for those classes in terms of grade? A? B?
This is my back-of-the-envelope calculation: in my university, one credit is corresponded to about 4-7 hours of work. So to barely fulfill my physics department requirement, it'll be 76 hours-133 hours, plus strongly suggested 2 classes, which should be 24-42 hours cover, say vector analysis, Fourier transformation, etc.
Of course, if you have already finished Calc I and II, then please subtract 40-70 hours from above.
And if you want to do some advance math classes, maybe add another 100 hours of class load?
This might sound intimidating, but keep in mind that you can spread them over 2-4 years. Which is not a lot.