If by modern physics you mean the standard, second year, survey course on relativity and quantum mechanics that most universities require then essentially you will need:
A intro physics background involving calculus.
Differential and Integral Calculus. (I also took multivariable calc concurrently with Modern Physics)
As for what is needed specifically from physics, I don't know if I can really narrow that down for you. Physics is cumulative and builds off what you've already learned. Everything you've learned in the intro courses will come up again eventually.