Buncha points from someone who could've graduated in two years, but took three, then went on to a PhD.
1) Great college life doesn't mean being in a frat. I was bored in high school. I loved college, because I went to a school with a very intellectual atmosphere and was surrounded by fellow geekily intellectual introverts. "Socialization" does not mean "drunkeness", and as others have said, the college academic experience goes well beyond coursework.
2) If you want to be an astrophysicist, then "how it looks on my resume" is absolutely meaningless. No one will care about your undergrad education when it comes to research jobs -- your research record in grad school and after is all that matters. Even if you move to industry, it won't appear, because your high school graduation date won't be on there (and if you put "BS in Whatever, 2011 -- finished in two years!", people will be rolling their eyes, not feeling impressed).
3) Again, if you want to be an astrophysicist, that means grad school. Rushing through the bare minimum to get the degree will make you uncompetitive for good grad programs. Upper-level electives are good, research experience is vital. Do not kid yourself into thinking that you can get good research experience while taking the kind of courseload you're talking about.
Seriously, many of us have been where you are -- bored in high school, taking advanced electives, and looking at just how much or how fast we can do things. You need to stop thinking about college as a competitive sport, and start thinking about what you want to get out of it and what it's going to set you up for afterwards.
And just as a completely random anecdote that proves nothing, I had an acquaintance in college who triple majored in technical fields and finished in two years (he came in with a lot of credit and those three majors had a lot of overlap, but he still worked himself to near-death). He ended up taking eight years of bouncing around research areas at two schools to get his PhD, partly because he came through those crazy two years with no idea of what he really wanted to do.