Sorry, I know I'm a little late on posting to this thread and you're probably in college off studying somewhere (congrats!), but I'm posting anyway because I think so many of these comments are very misleading.
It absolutely does matter the standing of your school's physics department when you're applying. Part of finding a school that's "a good match" is making sure they offer classes that you will love and are interested in, or you will struggle significantly more than you would at a "worse/easier" school that only offers you classes you think are boring.
I made the mistake of picking a school based on many factors, but not taking into account the physics curriculum. I'm really interested in theoretical/ mathematical as well, and there are very very limited courses in this area, and they are only available for upperclassmen (which is why I'm transferring)
To answer the original question posted: in terms of small liberal arts schools (what I'm familiar with) Oberlin has a very reputable physics department, Grinnell has great sciences because Robert Noyce just donates tons of money to buy the science department cool toys, and Kenyon's sciences aren't great, but the physics department has twice the budget as all of the other science departments (again, because of donations specifically to them)
If this doesn't help you, I recommend casting a really wide net for schools you like, and reading course catalogs to see if the subjects appeal to you (also, in terms of research, profs USUALLY research something to do with what electives they're teaching, because it interests them, but you can always email and ask. At large universities, research may be bigger/ more interesting, at small colleges it will be easier to be a part of it)
Also, be sure to read the course catalog for subjects outside your major, because you want to make sure that filling your core recs will be bearable :)
Hope this helps