Actually the employment in the aerospace industry is very recently growing.
The problem with employment in aerospace is, it's a narrow field in comparison to mechanical, so there isn't as many jobs offered. Also the US over the years has been moving away from the design of new airplanes and thus the funding has been in decline. At the same time, we need aerospace engineers regardless of whether or not the field is in high demand. Aerospace had a bad employment period since there was no need to hire new workers. However, a lot of the engineers in the field are retiring and this is leaving a huge void to fill, therefore the employment is on the rise. It was actually in the news over the summer. So if anytime's the time to be an aerospace engineer it's now.
Personally though I say that going with ME is the best choice, since you can always narrow your field in grad school, and a lot of the specifications for jobs in aerospace and such are taught and covered on the job and picked up easily if you are already an ME. I chose ME myself since I was lukewarm about aerospace and have constantly found myself changing my mind from aero to nuclear, or even biomed. The point is, there's always room for change in ME.
As for the difficulty, a lot of my friends who are in aero definitely have some tough classes, I haven't taken them myself though so I don't know if they're harder or not. However, I can most certainly say Mechanical Engineering classes alone are very difficult, and my classmates and I can tell you we've never worked so hard.