What's important to you in the college experience? This is the US, so practically all schools will provide you with a solid education, luckily: just make sure the physics department at your school is existent and large enough, and make sure you find out how easy it is for undergraduates to get involved in research.
If you want a challenge, then maybe you should look into "top" universities: they tend to be tougher. Liberal arts colleges often have a reputation of rigor. Try Reed College or the University of Chicago for some challenging schools, both liberal arts and research university (Chicago's undergrad is structured like an LAC with a research university, really).
But you're going to find academic difficulty at many, many schools (just search around, you'll find out what schools are difficult: I sure did, and those are the ones I applied to!), so you'll need a way to distinguish between those. As mentioned earlier, what other factors are important to you in college? Do you want a liberal, conservative, or moderate feel? Do you want a city or rural? Do you want large or small? Do you want a more academically well-rounded school or one with a strong STEM focus? Do you want a school with a strong core curriculum (Chicago, Columbia) or one with an open curriculum (Brown, New College in Florida)? Does AP credit acceptance matter to you? Is it important that the school have an engineering department?
All those should help you narrow it down from there. I say apply to 3 schools you feel are reaches (some people apply to much more since chances are so slim at very top-tier schools), 3 you feel moderately confident you'll get in, and then however many safeties you feel you need. It's only a suggestion: I only applied to 4, so I'm not one to judge! I've heard of people who applied to 16, though I think you should narrow it down more than that so it's easier to choose once you get in.
And mgiddy911 is right: visit the schools before you decide which one to go to! My top choice shifted after a visit, much to my surprise. You never know what a campus is truly like until you visit.