I emphatically disagree with you and your friends about the value of a liberal arts/humanities degree. What you have learned is how to recognize patterns in thought and expression, how to read and write well (There's no end to education in these areas.), i.e. to express your own thoughts clearly, to employ logic in thought and expression. In studying philosophy, I would think, you were steeped in the history and analysis of ideas. I have college friends with BA degrees in international relations. Not one of them works in related fields. Yet they all have done quite well in careers in law and journalism.
Beware of the modern tendency to confuse education with training. Training is one variety of education, but training for a profession usually has narrow goals. You can be trained for nursing, for example, in nursing school. Other skills can be learned in those for-profit "colleges". Graduate and professional programs in universities focus on training students in fields like medicine, law, the sciences or teaching and researching, say philosophy or sociology. Taking courses like art history during graduate education in the sciences is very much frowned upon. You don't go this route to "broaden" your education; quite the opposite. Your last opportunity to indulge in study for curiosity's sake is college. I suggest that you take advantage of it. As an aside, some medical schools will accept some students from liberal arts programs, with some remedial courses. I'm not suggesting a career in medicine for you. Just an example of similar opportunities that may be out there.
To someone with an interest in science, especially foundational sciences like physics, the history and philosophy of science is a fascinating subject. If you haven't taken any courses in it, could you do so now? Just guessing, but you might be more interested in the intellectual content of physics than it's nuts and bolts, especially experimental physics. When you imagine yourself involved in a subject like particle physics, are you thinking about unified theories, Higgs Bosons, the nature of quantum mechanics and relativity theory? Or do imagine yourself building and running heavy colliders, telescopes, satellites, and so forth? Research in theoretical physics requires a facility with higher mathematics. Given the nature of your academic work, I wouldn't think you are one of those who have difficulty with abstract thought, but that's required too. Experimental physics requires less advanced math, but instead, perhaps, the ability to understand processes, the consequences of manual manipulations of concrete objects and systems of same. Engineering, in other words. Another aside: When Robert Oppenheimer began his graduate studies, he began with experimental physics and was flunking it. Since he initially planned a future in experimental physics, he became profoundly depressed. He recovered when he began anew in theoretical physics. Went from Cambridge to Gottingen, I believe.
Don't become hung up on false dilemmas. You could always attend a community college for remedial studies of math and physical sciences while you work part time. In my college town, there are lots of baristas, cabbies, etc. who are doing just that. Some are able to save enough money to travel and do other fun things, so I imagine you could make enough to pay tuition at some future date. On the other hand, you could become academically rusty if you dwell in the "real world" too long. After a full-time student's career, you can be amazed at the sorts of skills learning requires (like listening, thinking, and writing at the same time). True in my case, anyway (I'm retired.) If I had to take one-hour tests in the chem courses I tutor, I'd surely flunk out. Working while studying will take more time, and may require more self-discipline than full-time studies. But if you're sufficiently passionate about physics, you should be able to hack it. You might not have to live at home if you share living spaces with roommates ( Grad physics students,? You watch "Big Bang Theory"? On second thought, maybe not. ) Don't worry too much about how an unorthodox career path like this will look to graduate admissions officers. You had a change in heart, in your junior/senior year, completed your degree and then went back for a physical sciences degree. Financial resources necessitated part-time studies. They've probably seen successful students who've done that. But they will probe your motivations. Be clear about these; be prepared. If accepted to a grad program, you'd have to come up with graduate tuition without outside work, but there are teaching assistantships and other ways to pay part of your expenses.
Given all your uncertainties, you would probably benefit from talking with people in the field, in order to get a rough idea of the nature of a physics career. On the other hand, don't blindly accept everything they may say. Stay open to a diversity of views. Note the plural "people", not "someone". If professors and so forth are too intimidating at this stage, have a chat with grad students over a beer or coffee.