I don't know about MIT, but I tried doing it for a while and realized that the teaching style for engineering classes and physics classes are very different, and consequently, that EE was not for me.
Engineering classes typically focus more on methods and application. For example, in a typical circuits class, you will encounter all sorts of different types of circuits and they will give you equations and transform methods to solve them. There usually is less emphasis on how the stuff works on a physics basis.
Physics courses are the opposite - more theory and (somewhat) mathematically rigorous, but with less emphasis on application. Now you asked for how tough it would be, and I mention all of this because it will depend on where your interests lie. I started off as a physics major with an interest in EM and state machines. However, I found the EE classes very tedious with lots of homework doing the same tiresome thing over and over. I was use to (and preferred) the shorter physics assignments that would take hours to do and required a lot of sitting and thinking. Granted, I didn't get terribly far in the EE curriculum before I opted out, but I hope that makes sense.
If you're originally into EE and wanted to add physics, then by all means :) I'm not above shamelessly promoting my major.
As for course load - it'd definitely possible, and depending on how much AP credit you have, it may not be terribly cramped. Note however, that since you'll only have so many semesters and every college has different requirements for each major, you will probably have little time to take advantage of your school's humanities/social-science classes (all science and math is not all it's cracked up to be - believe me!).
In general, you'll have to set up the course schedule for your 4 years by the end of your freshman year. It can be quite a burden, especially if you're not sure how difficult courses will be or if they'll conflict. Then there's always the possibility that you schedule a semester that's too tough and need to drop a class.
Ok - so I've gotten slightly off topic. The moral is - double majoring is more difficult than single majoring. Double majoring physics with EE is more difficult than many double majors. The specific difficulty depends on your ability to manage your time and how you respond to the classes. It's not impossible, but it locks you into a specific schedule, so make sure you want it or you'll waste a lot of time when you could be doing research, co-op, or a different double major. If your school offers a dual major program, you may want to consider that. Make sure to talk to your advisor.
(I assume that you haven't started college yet.)