Here's my take on the astronomy/astrophysics distinction:
https://www.physicsforums.com/showthread.php?t=121921"
Nowadays, astronomy/astrophysics really is just a branch of physics. In most undergraduate programs, you're expected to take all of the core physics classes and some of the advanced ones. There's a lot of variation in the graduate programs, but to really be competitive on the job market, you'll usually need to go to one that gives you a lot of training in physics.
The training you'll get as an aerospace engineer, although often very technical, likely wouldn't involve as much physics as you'd need for astronomy. However, astrophysics departments will often hire aerospace engineers to help them build and design their instruments. There are some astronomy faculty members that are primarily instrumentalists, but they usually have astronomy or physics degrees.
If you're still at the point where you can take some extra classes and get a physics/astronomy degree (would need at least two years, I think), then sure, go for it. It's a leap, but not a ridiculous one. If you don't have time to get a physics/astronomy degree, then you would need to demonstrate physics competence by performing well on the physics GREs. Good recommendations (emphasizing your quantitative abilities) are key as well.