I don't think you should do that and here's why:
First: Age is a problem.
I'm not here do give you false hopes saying that despite being 25 you could still revolutionize some field in physics. That's not impossible, of course, but that's improbable. 25 is not old to do physics, 25 is the exact age at which you should do your best work... as a PhD. By 25 you should already have gotten your PhD. The lifespan of a physicist career is 8 years 10 years tops, between 25 and 35 years, when you are fresh from grad school and spend most of your days in the lab or staring at a black board full of equations. After that, if you did everything right, probably you will have already done your best work. Some papers advocated that 25 to 35 years your brain will be at it's peak, when you have an already develop brain (out of teens years) and not so old that your brain will start to get "old".
To start an undergrad course in physics at 25 you'll finish your studies probably at 35-40 if you manage to finish, because I believe that you intend to have a life, to get married, buy a house and have kids which you won't be able to do if you're still in school.
So, if you decided to do this, do be careful, age is a bigger deal that you might think and bigger than people who are trying to make you feel better and not to say the truth to you will make.
Second: Field of study.
You are a medic, you probably just have an undergrad degree, which is a lot consider that you do medicine. In most of undergrad courses of medicine you won't even scrap the needed content to which a physicist undergrad will need to keep going to good grad school. What I'm trying to say is that everything you study up until now will be mostly useless if you decided to do physics. Are you really ready to give all the hours that you spend studying medicine up? Consider this carefully.
Third and last: Physics is not a mundane course.
You may call me presumptions and most of people here don't like to say that because of modesty or another reason, but physics is not a common subject to study. Physics is the mother of every science in the world. Even math is a tool for physics. Physics is in the roots of the world as we know it. And because of that physics is a really hard thing to study. If you forgive my lack of modesty, I can say that I'm a genius and I still think that physics is hard to do. I'm member of MENSA. What I'm trying to say to you is that physics may seem fun and interesting from the outside, but she is hard and will not play fair with you. She'll pull the rug on you and throw in the ground if she have the chance. She'll fascinate you, as she have done with millions of others, but that doesn't mean she'll be sweet and care about you.
She'll demand every waking and something sleep hour of your life. She'll take every piece of your brain full of caffeine and turn into knowledge. There's no middle ground. Either you life for her or you quit her entirely. I might even get to the point to say that physics, as a whole subject of study is the most hard thing you could ever study in your whole life.
But, if even then you still want to do physics, do it at your own risk and good luck, my friend. Maybe you'll meet in the field.