You can absolutely go from physics to EE. I went from physics to EE, and didn't need any catching up at all. In fact, EEs in my same year are catching up to me. I did, however, have a larger math background than was required of the standard physics degree, such as an advanced linear algebra course, that helped.