Well, by your definition of harder, I'd say they are both equal. EE's, I would say, have more work to do in general. EE programs make you do a lot of report writing, lots of big group projects, etc. However, physics, as I've noticed from my friends who are physics majors, requires a lot of commitment. Both are difficult, but I've noticed that EE's are just constantly being slagged by work -- not all of it technical.
I have been told that EE is harder, however, it is only due to the massive amount of specialization work that takes place. Many EE's at my school often tell me that the only thing they are really taught is how to solve standard problems and being able to apply this to solve problems presented in design.
being an EE, i would guess that learning physics to the corresponding level that i have in EE, would be harder. the advanced concepts in Electrical Engineering don't seem to me to be as abstract as advanced concepts in Physics (such as general relativity, standard model, string theory). i think the math in EE (as well as classical physics) is closer to 19th century math whereas some of the concepts of advanced modern physics uses more abstract 20th and 21st century math.
Being a double major in EE and physics, I think I am fairly qualified to respond in this thread. I will say that overall, I have been challenged an equal amount by both my physics and engineering courses.
People often say EE (or engineering in general) is not as theoretical and abstract as physics, and this is absolutely wrong. There's tons of theory and abstract subject matter in engineering, especially EE (for example comm sys and control theory, or solid state theory). The theory does not necessarily make it harder or easier though, since design and applied problems are usually just as hard as theoretical problems. At least at the undergraduate level, the theory and mathematical sophistication in EE is about equal to that of physics. I've actually probably used a broader range of math in engineering than in physics. In physics, we of course use lots of vector calc/multivariable calc and diffEQ. This is also used extensively in EE, but in EE complex analysis is very frequently used (for instance, in control systems), which is not used nearly as much in undergraduate physics. At the graduate level, this is likely to be a much different scenario.
Sometimes engineering courses are harder than physics courses because in engineering courses many things are just assumed to be true and are not rigorously proved or demonstrated, which can make learning the subject rather frustrating. However, sometimes this can make the engineering classes easier than physics classes too.
Also, I have found physics textbooks are, in general, better written and more readable than engineering textbooks.
Others have stated that there's just more work in the engineering curriculum than in the physics curriculum. I agree with this, but my engineering professors are a little more lenient with grading than my physics professors have been. Data analysis/uncertainty analysis and things like that are scrutinized much more closely by physics professors than by engineering professors. For instance for engineering labs, saying the result is 5 volts is usually sufficient but for a physics lab, you must say 5 +/- .5 volts, and you must justify that uncertainty.
I have to agree with leright. As an EE and math double major switching to physics and math double major, I find that there seems to be more work in the EE curriculum. The reason that caused me to decide to switch however, was the ridiculous amount of assuming things to be true and general lack of the exploration of theory. I found my phsyics classes within the EE major to be the most interesting and gave me a much better feeling of understanding than the EE classes. Even though in the EE classes I was getting some of the highest few A's awarded and usually doing very well on the final exams, I never felt that I had a deep, intuitive understanding of the subject matter, so I was never satisfied.