If you want a PhD in Physics (or any other discipline outside of medicine), you will need to master all of the elementary parts of the field. Perhaps you already have this knowledge, but if not, you will have to start at the beginning. You might find a school that will admit you to the graduate program with the provision that you take certain prescribed "make-up" work to learn the physics and math that you may be lacking.
By "medicine", do you mean earning your MD? Why would you want to go through all that work and clinicals and residency, only to turn around and spend many more years working toward your PhD in Physics? What is your end goal?