Lolgarithms, this is an excellent question. I would say that beyond the standard sequence of four semesters of calculus plus linear algebra (which is usually included in differential equations), nothing is absolutely essential. In fact, nothing beyond second semester calculus is used all that frequently. Oh sure, I've had to conceptually understand multivariable calculus, but the last time I had to parametrize a curve or surface and explicitly calculate a line integral or surface integral was way back in my freshman year of college, when I took multivariable. Most of the advanced math you need for physics will be taught in your physics classes. Most physics departments also make undergrads take one advanced math course. Beyond that, you don't really need any additional math. I took a lot of math, but only because I was also a math major.
If you go to graduate school to be a theoretical physicist, you'll likely need a couple of additional math classes. At that point your advisor can tell you what you should be taking. If you're in astrophysics he/she might recommend Fourier analysis. If you're in condensed matter, it might be linear algebra. If you're in high energy, it will probably be group theory (but many physics departments also offer their own group theory class). If you go to grad school to be an experimental physicist (that's what I'm doing), you don't need any additional math, though a statistics class might help. Most physics departments make their first years take a "Math Methods for Physicists" type of class, but that's it.
So long story short: take your four semesters of calculus plus whatever else your department makes you take, and you should be good. I'd certainly recommend taking additional math just for the fun of it, but I don't think it would help you much in physics.