One of the things about being young is that almost everything you believe you know is wrong. There's a pretty good chance you'll find out that other things interest you more than astrophysics. So having quality people and departments available, even in areas that don't currently interest you, is important. Dismissing this is a sure sign of, well, being young.Specifically, I wanna be an Astrophysicist, ...
As for going to a lesser regarded institution, there are many advantages to being a big fish in a small pond. If you're good, then you'll get lots of attention from professors. When you ask if you can work in their labs, they'll say yes. When you ask if they would be willing to supervise a reading course on a topic for which no class is available, they'll say yes. Take advantage of these things and everybody will be happy.