Being a double major in EE and physics, I think I am fairly qualified to respond in this thread. I will say that overall, I have been challenged an equal amount by both my physics and engineering courses.
People often say EE (or engineering in general) is not as theoretical and abstract as physics, and this is absolutely wrong. There's tons of theory and abstract subject matter in engineering, especially EE (for example comm sys and control theory, or solid state theory). The theory does not necessarily make it harder or easier though, since design and applied problems are usually just as hard as theoretical problems. At least at the undergraduate level, the theory and mathematical sophistication in EE is about equal to that of physics. I've actually probably used a broader range of math in engineering than in physics. In physics, we of course use lots of vector calc/multivariable calc and diffEQ. This is also used extensively in EE, but in EE complex analysis is very frequently used (for instance, in control systems), which is not used nearly as much in undergraduate physics. At the graduate level, this is likely to be a much different scenario.
Sometimes engineering courses are harder than physics courses because in engineering courses many things are just assumed to be true and are not rigorously proved or demonstrated, which can make learning the subject rather frustrating. However, sometimes this can make the engineering classes easier than physics classes too.
Also, I have found physics textbooks are, in general, better written and more readable than engineering textbooks.
Others have stated that there's just more work in the engineering curriculum than in the physics curriculum. I agree with this, but my engineering professors are a little more lenient with grading than my physics professors have been. Data analysis/uncertainty analysis and things like that are scrutinized much more closely by physics professors than by engineering professors. For instance for engineering labs, saying the result is 5 volts is usually sufficient but for a physics lab, you must say 5 +/- .5 volts, and you must justify that uncertainty.