I think now a days you are completely correct. While I am not a student of the history of Physics or Mathematics I have the impression that earlier in the 20'th century Mathematics and Physics used different formalisms in some areas although they were talking about the same mathematical structures. A famous but perhaps apocryphal story is that CN Yang was talking to James Simons about his research and Simons said 'Oh. Your talking about a connection'. I would guess that he was talking about a connection on a principal Lie group bundle. Here is a quote from CN YangI just do not understand how theoretical physicists can prefer using one mathematical tool over other one. Problem dictates mathematical tool.
"The beauty and profundity of the geometry of fibre bundles were to a large extent brought forth by the (early) work of Chern. I must admit, however, that the appreciation of this beauty came to physicists only in recent years."
— CN Yang, 1979
Simons was a student of Chern's. Chern was a Differential Geometer and his early work on the geometry of fiber bundles I think was largely done in the 1930's and 40's. Chern has a paper on the mutual recognition by mathematicians and physicists that they were both talking about connections on principal Lie group bundles.
Personal opinion: I think one of the morals is that there is a unity of mathematics and physics. IMO the idea that mathematics is just a tool of physics is passe at best. It reminds me of my sister's ballet teacher who viewed music solely as accompaniment to dance.