There are a couple other things I should add.
I can't vouch for this personally, but one of the reasons I made the mistake of getting a math PhD was that a topology prof of mine talked about how he had been to conferences where physicists and mathematicians interacted with great benefit to both sides. So, if that is true, there should be some benefit to being able to communicate with mathematicians and learn their language and culture, and to becoming a bit of a mathematician yourself, even. That's debatable, of course--maybe those guys were just string theorists or something, I don't know. As I said, this point of view is what lead me to the mistaken belief that I could still be in touch with physics if I did a math PhD. At least for me, that wasn't really the case--learning physics just ended up slowing me down and hampering my progress in math (of course, physics was an end in itself, so it was not hampering my own goals, but was quite at odds with what my adviser wanted from me). Of course, it could be that if I had stayed the course, eventually, it would have paid off, and I would be able to act as a mathematical consultant to physicists, as my adviser was, ironically.
Another point I can add is, I think, an unimpeachable argument in favor of taking more math classes, at least under certain circumstances. That is the story of how I learned linear algebra. I took linear algebra in the summer after my first year in college. I wasn't very good at math back then. I didn't understand it deeply, and I didn't retain a whole lot of it. Guess where I actually started getting it? That's right, my real analysis class. To some degree, the same was true of calculus, especially sequences and series. I didn't really get it when I took calculus (I was fine at doing integrals and derivatives and so on, but I didn't really understand why Taylor series are the way they are, why any of those silly convergence tests and error bounds worked, etc.), and then when I did analysis, I got it. And those things are useful for physics. You can be as dismissive as you want about being rigorous and all that, but the fact of the matter is that I didn't really get it, rigorously or non-rigorously before I took that class. So, the point is that when you learn a subject, you're not just learning that subject. It can also be an excellent way to review its prerequisite subjects while learning something new. And I think part of it was the insight of the instructor, which I may not have encountered elsewhere. I think it also had to do, not so much with the rigor, but more the stage before the rigor, where I would have to come up with all the intuition to inspire my proofs. Now, of course, some people may have learned linear algebra very successfully elsewhere and so on, and in that case, they would be fine not taking that real analysis class. But if you happened to be like me at that stage, it would have been very beneficial, and in general, I think it can give you a boost to your problem-solving skills. I think it's just too simplistic to boil everything down to direct applicability.