I just started, but I can give you some information on how I chose where to apply, and how I decided where to go.
Realize that, when pursuing a Ph.D., you will spend a substantial (4-7 years+, depending on a number of factors) amount of time at that school. So, yes, it is important to apply to and attend somewhere that you think will be a comfortable place where you enjoy living for that duration. If you have had the fortune of traveling around, maybe you have an idea of where you would like to live, and where you wouldn't. I have had this fortune, and so I had a pretty good idea of where I would like to live, and where I wouldn't. I also wanted to use this time to live somewhere quite different from where I grew up. Weather is important, because it's something you will have to deal with. So, of course, location, weather, available activities in the vicinity, ease of traveling to and from the area are all important considerations when choosing where to apply, and, ultimately making that decision on where to attend.
That being said, deciding on graduate school is far from just deciding where you want to live. You have to make sure there are researchers at these potential universities with whom you would like to work. Funding is also an important consideration, as is average time to completion, which can certainly depend on department politics. As a graduate student, you will be spending a very substantial amount of time at the school working, and less time doing other things in the vicinity, like you may have in undergrad.
Unfortunately, choosing a graduate school is a many-variable problem, and you have to do your best to optimize all of these variables. (Unless, of course, you only end up accepted into one place. This was the case for me, and it greatly simplified my final decision. Deciding where to apply was the most time-consuming part of the process for me.) If you find a school that isn't "ivy league", but has active researchers in your field, is in a location in line with your preferences, and you have good financial support, you have a good impression of the folks there, I say to go for it. School visits can hopefully bring a lot of these factors to light.
As for happiness, well, if you're studying in a field you love, and you like doing research in that field, then, yeah, I think you will be happy. I am happy. Busy, but happy.