Sure, it will. The question is what you really intend to achieve. Equipped with all mathematics which is necessary, you can certainly study particular fields of physics and understand them. However, that does not mean you will get any relevant degree in physics, you just get knowledge and only in that particular field of physics.
But there is one thing you should know: Yes, the language of physics is mathematics. Nevertheless, physicists and mathematicians speak different dialects of this language, and a large part of a regular study of physics is to learn this dialect. This usually needs time and a lot of practice. So if you only study a particular field, and know the mathematical mathematics, you will open a book and find a strange dialect. This has to sound funny to you, but to study a certain science has more to do with learning a language than most assume. E.g. lawyers use our common language in quite a different way than ordinary people; look up some contract and its salvatory clause:
That's not how ordinary people speak, yet it is ordinary English. It is similar with physics and mathematics and the usual way to study physics gets you used to it.
If you only want to understand, e.g. general relativity, then it won't be necessary to learn thermodynamics or electrodynamics; a good portion of differential geometry might do. However, if you want to take part in discussions why general relativity and Newton's gravity are different, electrodynamics might be useful as it is a frequently used comparison. If you want to understand the physics of black holes, then thermodynamics is necessary.
So as it is always the case with questions like yours: It depends on where you want to end up with your path.