i might recommend Physics/Math as an undergrad major, take the necessary Calc, Diff Eq that are in common with both Physics and EE, take additionally Complex Variables, Probabiliy Theory, Matrices and Determinants as whatever math electives that you can apply to your Physics/Math degree, take maybe 3 or 4 junior level EE courses that you will need to get into a graduate EE program (but you won't need to take EM fields in the EE dept.) and then, with most schools, nearly all of those senior level EE courses, you can take for graduate credit as a grad student (as long as you take sufficient grad level course for credit). when you get your Masters, it could turn out that you might want to continue EE (for a PhD) somewhere else (or you'll get accepted only someplace else). usually you will get a year credit for the PhD program for your MS, but if you did EE the whole way, there might be significant overlap between the two grad programs, even if one was for an MS and the other for a PhD.
i'm projecting; it's sort of what i wished i did.