I have a PhD in physics. I have never taken a course in differential geometry nor group theory. I did just fine.
Doing just fine is hardly the point. Some people have different learning styles and preferences. If you have a more mathematical mindset, a couple more math classes might help such a person. If you check my background, I have a PhD in math, but from talking to physicists, there are certain very small parts of their field that I seem to have a better grasp of than at least the typical physics grad student, and not just from a more rigorous point of view, but I actually have more intuition than they do about those things. For example, Clifford algebras, which come up in dealing with fermions. Another thing that physicists might profit from that they don't seem to use too much is differential forms. It's not necessarily stuff you would cover in a standard course, though. Sometimes, differential forms are covered in differential geometry. But often, they are so badly taught that it defeats the purpose (for a proper treatment, one would need to turn to Arnold's book or A Geometric Approach to Differential forms). For Clifford algebras, too, if you learn it from, say, 90% of the books out there, there wouldn't be much point because they explain it so badly (decent treatments may be found in Penrose's book, The Road to Reality, for example). Even with the good books, you still have to think pretty hard for yourself about it to get what I consider to be a reasonable level of understanding.
It's easy for a physicist to think my approach is over-kill, but for me, it is certainly not. It's the way I have to do it, and if I have to do another way, I'm not interested. If you are like me, my brain just rejects certain things, until it can get more intuition. For example, the Dirac equation when I first saw it. I went and thought really hard about Clifford algebras and read lots of stuff and my brain no longer rejects it. Then again, if you are like me, you might not do very well in academia, so take it with a grain of salt. I'm now looking for work in industry, I think largely because of my insane compulsion to think too hard about stuff that is explained badly, just because I am offended by it, even if it isn't particularly relevant to my research.
You don't really need to take classes, though. You can study on your own because then you don't have to cover a whole course worth of stuff. The approach I would recommend is looking into more math on an as-needed basis, not necessarily dismissing it, but not spreading yourself too thin.