Mathematics and physics are closely related and specialists in one of them have normally good knowledge about the other one. However, I think the two are different in a major point: It is all about coordinates in physics, and mathematicians hate coordinates.
This is of course exaggerated and not always true. There are parts of mathematics which do require a similar way of thinking, and parts of physics which deal with principles and laws rather than frames. But at its core it is true. Physics is descriptive and in order to measure and describe something you need coordinates. Mathematics has of course to provide the techniques to deal with different frames, but its not central, and I have met many mathematicians who were proud to be bad at calculations. E.g. if you have a strict upper triangular matrix, then this matrix is nilpotent: the longest diagonal vanishes with every multiplication. This can easily be seen by calculation. A mathematician, on the other side, might explain this phenomenon with a nilpotent ideal of a ring. So if you see the coordinates of that matrix, then you're closer to the thinking of a physicist, if you see the ideals, then you're thinking is closer to that of a mathematician.