The math you need depends on the area you are going to study.
For example: Fluid mechanics is going to really work your calculus, up to at least multi-dimension partial differential equations. Probably you will want to know about curvilinear coordinates. And possibly you will want matrix algebra. And the computational parts will get you into a variety of algebra, discrete math, and finite difference or finite element mathematical methods.
Electronic circuits may cause you to want some topology. Maybe quite simple topology such as introductory graph theory. It will also likely cause you to want some formal logic.
Crystallography will likely get you into some interesting geometry, group theory, and again some matrix methods. And again probably some computational methods.
If you want to study general relativity then you want differential geometry.
And if you are doing quantum mechanical anything, you are going to be getting into matrix algebra, group theory, topology, computational methods, and just loads of other things.
Math is a tool. You should try to get the best tool set you can manage. You probably need to focus on the areas you are specifically working on rather than studying all the math you can find. Unless you are mutant smart. So look at the course catalog for your school and find the courses you are interested in for later years. Look at the math they use. Plan ahead and get the tools you will need. But don't close doors you can keep open.
Example: I mentioned group theory for crystals. But it is often useful in any situation in which you find symmetry. And symmetry exists in a large variety of problems in physics. So knowing group theory, at least a term's worth, will give you one more quite powerful tool in your tool box.