fasterthanjoao makes a lot of good points. I always thought quantum was going to be so cool to take after all the popular science accounts of it, but the class is really nothing like the books: a lot of math. The only really interesting parts of QM when I took it were when we talked about Bell's Theorem, Uncertainty principle, and very briefly, Noether's Theorem. And you can get the gist of these ideas on wikipedia or other books. Finding the solutions to the infinite square well or doing perturbation theory wasn't THAT bad, but... also wasn't all that exciting for me.
Really, I just find the concepts of QM very interesting and philosophically profound, and you don't need all that math to know something about them. Books like "The Elegant Universe" or "A Brief History of Time" is a good place to start.
I agree with fasterthanjoao that intermediate E&M is pretty cool though. For some reason the math in E&M sort of "clicked" with me (lots of multivariable calculus).