I think it's essential to self-study something. Eventually, you don't have anyone to teach you or grade your work, and you have learn to deal with that. It's a pity to have the mindset of having to have something to show to employers or grad schools for everything you do. I want to have something to show to myself first, employers second. True, I may be in a tough position career-wise because of this at the moment. But I know my priorities. Math first (pursuing my own mathematical curiosity, that is), jobs and publications second.
As a former EE student, I can say that trying to learn quantum mechanics and solid state is pretty daunting. I was really annoyed when I took electronics because of the shallowness of the way the physics was discussed. But, I wasn't quite ready for quantum mechanics. I made some attempts, but didn't get far. If you are well-prepared, it could work out well. It helps to know a lot of math (most importantly, linear algebra), and as an EE major, I wasn't that mathematical at that point (now working on math PhD). It also helps to know a lot of physics, particularly Lagrangian and Hamiltonian mechanics. I am pretty extreme about having an in-depth understanding, though, so maybe it is possible to get by with less background.