I'm a double major in math and physics and I'll be graduating in the fall, so at this point I've thought about the choice quite a bit.
First some context. I started on a slightly different path - I started as a physics major and took 1 math class per semester out of interest. I had plans to go to grad school for physics. Eventually I took the first abstract algebra class, which was by far the most eye-opening and mind-blowing class I've ever taken. This convinced me to go to grad school for math instead, and since I only had one physics requirement left at that point I decided just to keep the physics major.
I actually feel like my math studies have benefited a lot from the physics classes and vice versa. It is probably true that I would be better at math now if I had only taken math classes, or better at physics if I had only taken physics classes, but having a reasonable amount of knowledge of both has given me more than one point of view to look at things. For example, I'm in my last physics class now, electrodynamics using Griffith's textbook, and since I've done graduate level coursework in topology, I have been thinking about the material in terms of differential forms, exterior derivatives, the generalized Stoke's theorem, etc. Because of this, most of the derivations in the textbook are very easy for me to read since I can see that it is just repeated applications of the very basics of the theory of differential forms as opposed to several different theorems that initially appear unrelated. On the other hand, I'm also currently taking a class on operator algebras, and since I've had a year of quantum mechanics, everything in the class already has a lot of motivation for me. Some other students in the class without any physics background have had a difficult time seeing the motivation for a lot of the concepts and it is always more difficult to learn math when you don't see the motivation.
There's a good chance that I won't study anything physics-related in grad school, but I do not feel that studying physics has been a waste of time at all. For the other way around, i.e. planning on going to grad school for physics instead, I cannot say as much since I know less about it. What I can say is that theory can involve some pretty hardcore mathematics if you want it to - I worked for a mathematical physicist last summer, and ended up doing a lot of algebraic topology, K-theory, functional analysis, etc, none of which you would typically learn as a physics undergrad, and if I hadn't had a strong interest in pure math before going into the project then I do not think I would have gotten the opportunity to begin with simply because most physics undergrads do not have the background to work on such topics.