To begin with, I think that:
Many physicists are also excellent mathematicians, and many mathematicians are quite good in physics.
As a mathematician, I could witness that when I was learning at the university, there were many physicists in the class learning with us topology, group theory etc.
These two disciplines always gone together. At the beginning, there were no mathematicians and no physicists, they were most often both of them. Archimedes, Fermat, Descartes, Pascal, Robertval, Newton, Lagrange, Legendre, Gauss, and many, many other. In a letter of Abel, I could read once: "I'm the only one in Europe that deals only with mathematics".
Admittedly, by our days, you have to specialize in a sub-sub-sub discipline of these sciences.
Nevertheless, the links are sufficiently strong to allow mathematicians to contribute to physics, especially fundamental and original ideas (that's the domain were we are really good after all).
This being said, in addition to the reply of Vanhees71, I would like to give an example at a very, very elementary level, where most physicists are unaware of a hole (here I say most and not many). I mean the notion of capacitance matrix. When I read once elementary books in EM, I immediately felt the hole (I call that my mathematician intuition). Then I asked several physicists (even in this forum if my memory is good), and they reply to me that for them, it is evident that the charges of the conductors depend linearly on the potentials, invoking the "linearity of the equations of EM". After all, that's what is written in Jackson, Landau and Lipschitz etc. etc.
Then I tried and succeeded to answer by myself to the question for conductors in free space, which involves a uniqueness theorem by no way trivial, that I found in some lectures by Dr Konoplisky. I couldn't believe that this is not dealt somewhere, and I continued to seek in books. I finally found that Griffith was aware of the problem and of its solution (but he writes nowhere the solution), and that Jefimenko book is perhaps the only book that contains an (almost) valid proof by induction. Regarding Maxwell, I think his proof is incomplete (he proves only the multiplicative linearity, which is trivial, but not the additive linearity).
And all of that concerns only conductors in free space, but what about conductors in a mixture of dielectrics whose properties are described a tensor?
Only recently were two articles published on the question. Unfortunately, in some aspect, they are misleading (I am currently writing an article where I develop the theory in a general framework, even if it is an extremely modest contribution to physics.)
The point I wanted to illustrate is that holes exist even at such an elementary level. How is it possible that elementary physics has not been duly formalized, akin to what the Bourbaki school did in mathematics ?
I would like to cite the problem of microscopic to macroscopic EM field, where I feel that something big has been missed as well; but I am currently trying to study the books of Robinson and De Groot who have dealt extensively with the subject. So, I will abstain.
Edit: actually, Jefimenko does provide a valid proof for the elastance matrix, but assert without proof that it is invertible (its inverse is the capacitance matrix). Nevertheless, the tools he provides are sufficient to establish the capacitance matrix as well.