In some ways, experimentalists can be more constrained than theorists. A theorist can grab a laptop and go code at the coffee shop if they choose. An experimentalist is largely (at least partially) tied to the lab. In addition, I have many friends who needed to wake up at 2am (or some other godawful time) to check on an experiment.
But, I don't think physics is any different than other research jobs. Why would a biologist or chemist be required to put in less time than a physicist? In order to come out on the other side with a PhD (I am assuming here that physics research requires a PhD - not always true, but typical), you need to really enjoy and be dedicated to your research. So, work doesn't always have to feel like work.
You don't put all the hours in at research and writing grants, the years of grad school, the relatively low pay (compared to an MD, for instance), to do something you just cannot stand or think is just average. You have to love some part of your job to do that.
Does that mean you are going to not have time for anything in your life? Depends on the person. Everybody needs a balance in life. If you cannot find that, you will burn out. You see that in physics too. So, you have to learn to make time for your personal life - your family, friends, hobbies, sports, activities, etc.
For instance, in grad school, I made it a rule that I took Saturdays off from physics. From 8 am to 10 pm on Saturday - no physics. There were a couple of times I had to break that rule, but I usually stuck to it. I needed the mental break. It worked for me. It forced me to use my time better during the week. Everybody develops there own coping skills.