What is the difference between BA and BS in physics, in terms of course requirements, at your school? It varies at different schools.
If you're thinking about graduate school in physics, graduate school admissions committees aren't stupid. They will probably focus on the specific courses that you've taken, not the name of the degree. A BS is probably more likely to contain the courses that grad schools look for, but in the end it depends on what your school offers.
The small college where I was an undergraduate offered only a BA in physics, but it was enough to get me into grad school. The small college where I taught for many years offers only a BS in physics, but it's pretty much like the BA at my undergraduate school.
If you're going to be looking for a job right after your BS/BA, I have no experience with that route. However, from what I've read here, few employers look specifically for a BS/BA in physics. Instead, they look for specific skills: programming, data analysis, writing, etc.