I was in your situation X years ago.
Have you talked with Alan Wolf [not to be mistaken with Fred Alan Wolf]? He was in your situation X+Y years ago... (if I recall correctly) as the physics department was being dismantled.
He had some interesting stories about Cooper and UT Austin. At Cooper, Wolf (in Physics) and Kondopirakis (in Math) were my favorite professors.
( In fact, Prof. Wolf assigned an optional reading book for the Modern Physics class (214): General Relativity from A to B by Robert Geroch. I read it... it was interesting... but seemed verbose at the time... very few equations [written for the non-science major]. Later, as I was sitting in on Geroch's graduate-level relativity course, I re-read that book and it all clicked In fact, it's surprisingly deep in foundations beyond the typical "pop book"...I don't think I ever told Prof. Wolf about that.)
As a freshman with the same dreams, my plan was to get an EE degree first (like Einstein and Dirac) then go do the PhD in Physics. While I was already very interested in relativity, I felt engineering at Cooper was more practical and more marketable and Cooper was tuition-free at the time. I recall some advanced physics courses in the catalog that I had planned to take. I was disappointed to learn from Prof. Wolf that some of those courses are just on the books and had not been taught for years due to low interest [possibly a leftover of the physics program]. Silly me for not following up earlier.
After the EE circuits class gave me trouble, I ditched engineering and planned an exit strategy with with Prof. Wolf's help.
In my fourth and last semester, I took some outside classes at other schools (PDEs at NYU via the enrollment agreement [is it still active?] and Lagrangian mechanics at Columbia [as an independent-study class at Cooper]). Then, I left for Stony Brook to be a math and physics major. All of my courses transferred and I ramped up/overloaded on math and physics courses, fearing I was behind the regular majors--I don't think I was. Eventually, I did get a PhD in physics specializing in Relativity/Quantum-Gravity.
If circuits didn't give me trouble, I might have finished the EE degree. I'm not sure what I would have done at that point. Continue on in school for the PhD in physics or get a "real" job? :)
[Edit... one thing that I didn't realize that the time... as a graduate student in physics, you are likely to be supported by a teaching [or possibly research] fellowship, which covers tuition and often provides a stipend. So, more schooling along this track didn't lead to more student debt.]
Certainly things have changed since then.
Cooper isn't tuition-free any more. With Polytechnic now under NYU, I don't know if Cooper students can still take NYU courses.
If you can still take courses at NYU, you might want to take some intermediate math or physics courses (like mechanics or electrodynamics or quantum mechanics). I'd certainly consult Prof. Wolf for advice and options. Maybe there is a small group of students with similar goals and interest in physics or math. Maybe a special topics class can be arranged or a small group of students take courses outside. (It would have been nice to have a fellow student from Cooper in these outside courses.)