Engineering electives or double degree
I did EE for my B.S., and have somewhat regretted not taking any physics. I actually did run into a few closed doors where I worked that seemed to be locked except to those with the physics knowledge. Well, now I'm back in school studying physics, and I love it.
How long have you been studying EE, and do you like it? Have you taken any upper level physics? It really depends on what interests you, but, if you are anything like I was, you may not know, and it will just be a shot in the dark. I decided to study EE literally because I wanted to find out what the practical use of imaginary numbers was, and my Alg II teacher told me that EEs use them. The happy (part of the) ending is that I did find out a "practicle" use for it, but the real moral of the story is that, when I went to work, I wound up pushing a pencil (so to speak) pretty much all day long every day, while my education slowly deteriorated in my stagnant brain. I used to hate it when I was studying for my EE degree and people would naively say that EE and Comp. Engr. were basically the same. But, in retrospect, I would have wound up in the same basic job with a Comp. Engr. degree, and I probably would have been more prepared.
Anyway, I'm rambling. Everything above was somewhat personal, but glean what you may.
If you like working around imperfections to meet specifications (that are quite frequently motivated strictly by money and have nothing to do with stability vs. resonance or anything like that) then probably go for the electives.
If you like breaking ground and modelling and science, and yes, even philosophy to an extent, then I would recommend that you cut your losses this minute and go for the physics.