It is, but why do you want to?
You have to be dead set on the major you're going to take before you start, which is how a lot of students set themselves up for failure when they find the major they picked at 16 isn't what they wanted but feel obligated to press through. Most students change majors, it is an inevitable consequence of being exposed to the new topics and interests that are available to you in college. If you've got that grinding down on you emotionally, you will not be able to stay motivated enough to do the work to finish so early.
You will have to make sacrifices, possibly serious ones. Schoolwork is important and should be your priority above and beyond all else, but if you're in college then you miss out on a lot of things you'll never get a chance to do again if you don't make time for yourself. And some of those experiences are important "maturing" life experiences. And I don't necessarily just mean partying and having stereotypical "college fun", I also mean things like taking classes outside of the core requirements for your degree that broaden your horizons and sharpen your critical thinking skills, or electives within your major that show people (ie future potential employers and graduate degree advisers) that you have serious interests in physics and didn't just want to rush through doing only what was absolutely necessary to finish your degree ASAP. And success in any career can ultimately depend just as much if note more on supplemental skills like writing and public speaking than it does on the "core" of the field, going the extra mile to master those pays off in the long run. You will go much further in life as a completely average scientist who can lucidly communicate his ideas (and effectively convince people that you're worth funding) or speak in multiple languages or engage with people in a diverse range of industries and specializations than you will as even the best scientist in the world but who can do none of these things.
If you're talking about graduating early because you want to, not because you feel like you're rushed or think it will make you look smarter, then by all means go for it. But it's not like you have to or that there's any great benefit to doing so.