I just noticed I didn't vote. And neither of the two options given seems to be in taste. But if I had to choose particle physicists sure seems more relevant than a matematican.
I think of mathematics as a language that is very efficient and fairly non-ambigous when it comes to describing and communicating quantiative statements. But this doesn't replace a coherent line of reasoning that brings you to that mathematics. Sometimes I think mathematics can obscure the coherent line of reasoning. I often get the impression that physicists guided by language completely disconnected to the concept of a coherent line of reasoning. It's often motivated by that it seems to work. I never quite understood that philosophy.
To give my opinion of the question of the thread I think you at least need some kind of basic knowledge of mathematics or else the language barrier will obscure everything, but the language is partly a technical matter, and understanding the language is not the same thing as understanding what it tries to say.
I think the important things are how to find a mapping between mathematics and reality. Or the connection between the language, and it's dynamics/development, and whatever we are trying to express using this language, and it's dynamics/development. Sometimes physical theories are axiomatized and from there on treated purely mathematically. All you need to accept is that the axioms sensibly couple to reality. At least from a human point of view, I have hard to see how you can do that with without getting a little bit philosophical. And I think this can be the hardest part. A consistent mathematical formalism no matter how beautiful is not sensible as physical formalism unless it's axioms really have been chose in a reasonable manner. When we use the power of mathematics to draw complex implications from a set of axioms, "almost reasonable" doesn't seem satisfactory.
I think reality should guide us to building the appropriate language we need. Not let the "language purists" guide is in the quest of understanding reality, it seems to be quest driven by the wrong measure and hence I might doubt that the quest converges to the right thing. I guess it does boil down to what view of the scientific method one has.
Long time ago me a friend discussed (beeing students) what is the difference between what we know and what we don't knowthe best answer we came up to at the time was : time. It was based in the assumption that what I don't konw, I could (in principle; there may be exceptions of course) always find out, and "finding out" is a process that basically just takes some time. An implication from this seemed to be that, the point is not to know everything, the more important thing is to know how to get what you don't know when you need it, as soon as possible. This has been a guiding principle for me since. The more important thing I think I learned studying wasn't just what I learned, it was that I learned how to learn. And at some point you feel that you know enough basics, and also know enough to know how to further find out what you don't know. Another conclusion is that since time and memory is limited, "learning" things you never asked for may be a waste of time and memory.
I remember that looking at the history of science and the body of complexity it seems I need to first get a PhD in logic and philosophy, and then a PhD in mathematics, and they a PhD in physics, and perhaps a PhD in molecular biology wouldn't hurt either these days. One realizes that I would be hundred years old and still haven't mastered everything that might be remotely useful knowledge.
The only thing I really need to master, is howto efficiently find answers that make me grow, whatever the nature of these questions may be. The most useful answers often require first finding the good questions. These questions quickly inflates in complexity and requires good bookkeeping to not stall development. If I ran into a question I can't answer that is not the end of life as long as I can find the answer within a reasonable timeframe. What would be fatal OTOH, is to run into a question you can't answe AND not have the slightest clue howto find the answer. So knowing howto find out what you don't know is more fundamental than a priori knowing everything.
/Fredrik