I took Calc II along with another difficult science course during the summer, though Calc was two hours a day five days a week. The pace is very rapid, and you have to be sure you have time to work on homework every night. As long as you've got about 3-4 hours a day for each class (2 in class, ~2 doing homework), and the commitment to spend several weeks doing almost nothing but studying math, it can work. You simply must study every day, though. If you fall behind even one day, you can spend the next two-hour lecture completely lost without much flexibility in your schedule to catch up. It felt really nice at the end to have gotten those credit-heavy courses out of the way in less than half the time it would normally take.
I got a little bit tired of doing the same thing every single day, but I also did very well in those courses because you simply don't have time to forget what you've previously learned. Consequently, I didn't even need to study for the midterm or final--you're constantly exposed to the material. I know the human anatomy courses at my school (which are notoriously difficult and require large amounts of memorization) post grades ~15% higher during summer semesters compared to fall and spring, presumably for this reason.
TL;DR: if you can survive the monotony, taking just a couple courses during the summer can actually be easier than taking a more balanced load during fall/spring.