At my institution, an intermediate "Modern Physics" (after a three-semester calculus-sequence of intro science/engineering physics including mechanics, EM and Waves/optics/modern physics) wasn't required and was rather considered a "holder" course for students who didn't yet have the math background to move on to intermediate physics courses (like QM1, EM1, and classical mechanics). I declared my major a bit late, so I didn't ever take Modern, largely because of it's reputation as "filler"... and I found the intense physics coursework pretty nice to fit together.
Note that I did, however, have a pretty strong math background.. I had completed ordinary differential equations and linear algebra, I and was currently taking a partial differential equation/boundary value class. I think I was taking Quantum I, Mechanics I, Thermo, and EM1 all at the same time, along with the math class, maybe an advanced chem, and a few gen-ed's (it was, I believe my 22/21 credit hour term... oy vey, what was I thinking?)! While not easy, I had professors who for the most part set reasonable expectations (and I knew this by talking with other students in the department).
So think about your own math background, own talents, and talk to other students in your department about courses (both with regards to flow from one to another and expectations of professors). I knew 1-2 that had done killer-terms such as mine before (and skipped the modern course, which wasn't offered but once a year anyways), so I just decided it was my turn.