Maybe you're just getting cold feet and wanting someone to tell you that you're doing the right thing staying the course. I don't mean to trivialise your anguish (been there). I don't doubt your sincerity in liking physics. But you seem to have more passion for Classics, and have done for years, and are currently in perhaps the best place in the universe (whoever else is out there, I doubt they're studying Homer!) for Classics, and - rightly or wrongly - the place often thought of as the best university in the world. Whatever that is worth.
The humanities are a fine education. Classics is a fine degree. Oxford is a fine uni. Spending a few years of your life studying something you love under experts, surrounded by others who share your passion, is a tremendous experience. I don't know if I would throw that away on a last-minute panic about another interest, because you may or may not want a physics-related job down the line.
Remember: you do not need to wait for the permission of a university to learn something! There is a wealth of websites, youtube documentaries, tv shows, popular science books, science publications, journals, professional blogs, forums, textbooks, free online courses, astronomy clubs etc. out there that can, and probably will, more than satisfy your interest in physics.
P.S. Sorry, 'Oxbridge', not Oxford. But I think my point(s) still stand.