Been there...or almost. So this is going to be a long reply...(Nutshell: I am now an EE senior, applying to Physics grad schools for fall 2010.)
To "do" physics you will need an advanced degree such as a PhD. That will take at least 3-4 years I guess. So, spending an extra year in college to dual major shouldn't bother you. The timeline for graduate education followed by postdocs etc tends to be at least 5 years, so the longer duration should not hinder or affect your decisions. Consider that a price to pay for learning and then doing the cool stuff in physics that fascinates you.
My school doesn't offer a dual major, and I wish it did. But the Physics dept at my school is very supportive of students who are genuinely interested, and they allowed me to take courses in quantum mechanics, relativity and quantum field theory. You should do the same. Try and work on projects under physics professors. Your EE degree will not be an impediment to learning physics and in many cases you will actually see a strong connection (even in theory!).
I do believe that the EE degree coupled with a serious interest + coursework + projects in Physics is good overall academic package. But if you can dual major, nothing better than that! That is of course if you want to, because dual major doesn't mean necessarily better academically (this is contentious issue which I won't get into).
The fact is, there are many areas in physics where you can utilize your EE education and vice versa. Examples are quantum computing, solid state devices, nanoelectronics, photonics, accelerator physics, instrumentation in (all of!) experimental physics, etc.
You ask if there's an advantage. Well, there is certainly no disadvantage. At least some academic advantages are outlined above. But if you think its an impediment, you should do what your heart says. You can go to grad school in EE even with a physics degree. In fact many people do that in the US.
From personal experience, the biggest problem is convincing oneself and not doubting your decision. I don't know how Physics departments in the US look at non-Physics majors entering graduate school (I will soon find out I guess...lol), but I guess if you have considerable research exposure (something I lack) you should be fine. So whether you do EE or Physics, if science is what interests you, get as much research experience as possible.
Finally, to address your question (assuming you definitely do not want to double major), I can only say that there is no unique solution and no amount of advice from others will help. Just do what your heart tells you to. I am from India, and there are a lot of rigid rules in universities about who can do physics and who cannot, but schools in the US are obviously much more flexible and academically broadminded. So, do what you find interesting now. You can always do the other thing later. But don't let the time duration govern your decision if you want to do scientific research.
