Yes it is possible. However, you will have to take many remedial classes in your graduate program. It also depends on what discipline you're going into, ie. electrical engineering would likely take less remediation than chemical engineering.
My plan is to do electrical engineering so I am glad to read that. Is there a low acceptance rate into such graduate programs if I only have a physics degree and not an EE one?
Edit: Also, how does one take remedial classes? Can I take them as a graduate student? Or do I need to take them as an undergrad before applying?