Two things to consider:
1) Why are you attempting this?
2) Based on why then the question is will your health issue return?
Sometimes our body tells us things not to do but we have some kind of do or die belief that is driving us forward.
Astronomy and Astrophysics are noble subjects to master. However, the job situation is very bleak and only the best of the best get to the top ie survive as a research PhD. You can probably find out more in the APS site.
You may well ask who am I to say these things and so here goes. I got overloaded from college working on a BS in Physics and working first 20 then 30 hours per week pre-internet. I struggled to keep up but would often have to hand in homework late and partially complete. The problems I did solve, I was told, were well written expositions but because it was late I got docked a grade. Basically, when I graduated, I was too tired of school and didn't consider grad school at all.
Five years go by, I work at a company who will pay for a class at a time and so I entertain the idea of getting a Theoretical Physics PhD. I always liked General Relativity and Cosmology and the Unified Field Theory and I was ready to try again. My work as a Fortran programmer was boring and unfulfilling except for the easy money and toys you could buy. For me going to grad school was an escape to academia to explore new fields like computational physics (before the PC revolution too) and greener grass.
Each course I took was extremely hard as my math skills had eroded to such an extent that I barely knew Calculus and thought I knew Linear Algebra and Diff Eqns. How wrong I was to think that it would all come back. I had no support network, no PF and was still working so I couldn't just hang around the physics dept. However, I did have my Schaums Outlines Math Handbook and trusty sliderule and calculator.
On top of that, there was a ticking alarm clock that was the qualifying exam and competing with recently graduated undergrads who were on top of their math skills. I persevered through several courses until I realized that the qualifying exam was likely a no go unless I could stop working but I had a family and couldn't. There was also an undercurrent of profs who didn't like the notion of a grad student doing part-time study and being independently wealthy ie not at the poverty level. I guess they felt I wasn't easily controllable.
Finally, the straw broke and I realized that COmp Sci was a better fit and so I transferred my physics credits over and got an MS degree fairly quickly. I stiil look back forty years and think maybe I can try it again. However, I realize that I can't compete with aspiring undergrads and GRE tests to get accepted into my university's degree program.
I even looked into the secret alternative path to a PhD a prof told me about realizing that I couldn't compete academically but could wih work experience ie doing a disertation-worthy project, getting PhD sponsorship and working my buns off to get it done but got stuck at the "find a disertation-worthy project" at work.
Now I'm looking at retirement and hoping to live the life of a gentleman of leisure with a hobby in math, physics nad comp-sci. Haha! :-)
Bottomline, think of your circumstances, job prospects and where you want to be in the next five or ten years and let that be your guiding principle.
On another note, the mathispower4u.com website hosts a lots of math videos like 5000 or so that cover HS math (in case you need to review something), Calculus 1,2,3, Linear Algebra, Differential Eqns and Statistics. The big four of any physics undergrad curriculum. Each video covers a specific topic and lasts for ten minutes or so solving one or two problems at a time and allowing you to pause and solve and then see how its done.
Alternatively, there's Khan Academy with many more subjects and of course MIT Open Courseware but I find 60 minute lectures are tougher to listen to but that's just me and my ADD.
Hang in there and make sure you want this for the right reasons as you'll need it to overcome any health anxieties.