Rettix:
I am in your exact situation, almost. I majored in computer engineering, decided that I liked physics, and minored in physics because I was too far along to justify switching. I'm currently in the process of applying to grad schools...so I guess I'll let you know how I fare in March.
The sort of Electromagnetics that EE majors take is good, but it's not quite the same perspective that Physics majors take. Engineering focuses much more on application than theory. However, you can use that to your advantage: You have a practical view on such things that most physics majors probably lack. For example, having taken several courses on circuits and amplifiers, I seem to have a much better handle on circuit problems than my Physics co-students. In fact, I'd say that most circuit problems the Physics majors do are pretty basic in comparison to what any EE/CompE studies. But of course, there are other areas where a EE will be lacking; but that's why you added the Physics minor.
Have confidence that you can very likely get accepted to a Physics program somewhere; the question is where. If your eventual goal is to become a professor at a research institution, then you will have to do a lot of hard work and strategizing. But I think it should be within the realm of possibility.