It depends. I can certainly very difficult while you are a undergrad student, and it doesn't get much easier as a PhD student.
I have friends who managed to raise a family while doing a PhD, they were perhaps not working as many hours as me but they were usually much more efficient than me when they DID work.
That said, I am a pretty good example of someone who did not manage to find the time for a social life while doing a PhD; so it really depends on how you are.
However, physicists are not exactly unique in this respect, everyone -regardless of education and job- have these problems to some extent. But of course it is worse for those of us who goes into a "challenging" career (I suspect it is even worse for say medical doctors)
So the bad news is that you will never have as much time for a social life as you have now, regardless of what you choose.
Welcome to the real world