I think a lot can depend on the specifics of each individual program, but there's no reason to believe that in general it's harder or easier than other engineering programs.
I think the advantages of engineering physics lie in the fact that you get the professional qualification of an engineer when you come out. The course work and projects tend to focus on more "applied" physics (but that's a generalization).
The disadvantage is that you don't have as much opportunity to explore the different branches of physics as an undergraduate because you have to fit in a number of engineering courses. So if graduate school in physics is your next step, you might not get the same exposure as a more traditional physics student. On the other side of the coin, I'm not sure that engineering physics graduates have the same number of opportunities for jobs within the profession as the other branches of engineering.