You will find that both university algebra ( groups, rings, modules, linear algebra and so on) and university calculus are developed a long way beyond their high school counterparts.
To effectively study physics at higher level you will need both. There is some unifying theory as well, which helps when you have covered enough to draw it all together. In the beginning there is a huge amount of what seems like disparate material, but when you become more familiar you will see that there are many unifying threads and principles.
You will also find that, since algebra is inherently easier (or at least less laborious) than calculus, there are techniques, such as the Laplace transform, to reduce calculus questions to algebra ones. This is the opposite direction from your question. As Vanadium has said it is not recommended to try to go the other way.