I actually posted something similar myself, though my situation is a little different.
I am 34, so not much younger than you. I have always loved the sciences, but I never pursued them professionally (long story as to why, briefly explained in my own thread).
One positive thing that I can tell you is many adults go back to school in order to pursue other professions. I remember seeing statistics somewhere that showed 30% of all adults over the age of 28 will end up going to college or going back to college for a new career path. Though I feel those numbers are a bit high, I can tell you from what I've seen personally, there are lots of us doing it.
As far as physics, you can most certainly make a viable career out of it. Though it is a competitive industry, it is really all about what type of job you are willing to take. Also, are you willing to move across the country to take that job? You can almost definitely find something, it is just a matter of how picky you are about what you want to take.
Another positive thing I can tell you is that I have already started over the summer and the teachers and students responses to me have been great. I was a bit nervous about what the students would think of an adult like myself being in their classes. Once I started classes, I quickly realized the students look up to me because I ask the questions they want to know, but many are too shy to ask the teachers themselves. They also love working with me because I am more motivated than most of them at this point in their lives. The teachers have (so far) loved me because of my participation and motivation in class. Anyway, once you start college, you will find there are quite a few adults our age taking classes along with you!
The whole college thing is new to me, as my previous profession required a certification and C.E.U.'s (continuing education units), so a degree was more of a waste of money than anything else unless you wanted to pursue particular sub-fields in the industry. Though I have been a regional manager in the past, I was admittedly very nervous about not knowing what to expect. However, the experience has been amazing, and the feeling of finally pursuing something that I truly love is indescribable. If your heart is truly in this, don't let anything hold you back!
The only bad side that I can tell you is that going to school for physics is going to take a lot of time away from your family. Not at first of course, but as you get into the more advanced physics and math courses, you will need to study a lot. Everyone has told me numerous times that physics is one of the most difficult majors and that there is a vast amount of knowledge you need to soak up. So, there may be some times in your life where school will have to come first and family second. Just something you need to consider before jumping into this.
As far as what Dave Aioli said, I think it is all great except one thing I was a bit confused about. When Dave stated that many of the older students end up dropping out of grad school because of family and financial issues, one question that I have is what about stipends? From what advisors have told me from several schools, about 70-80% of students getting their masters in physics end up with stipends, and 95% of students getting their doctorate in physics end up with stipends. So I would have to ask that once they've gotten through to grad school, how is money a hindrance when they can earn money while pursuing the degree?
Anyway, I wish you the best, and hope that my personal experiences have helped alleviate some of your concerns. If there's anything else you wanted to ask or talk about and don't want to post it on the forums, please feel free to PM me any time.