Yes, and I'm allowed to say that because I switched from EE to math, myself.
I want to go to grad school and I know how grim the situation can be for pure math, so I guess I'm looking for people to tell me otherwise or reassure me...
Not much reassurance for me. It's pretty grim. I can say that grades tend not to be a big deal in grad school, so the coursework isn't that scary, even though it is pretty hard. The way I see it, qualifying exams aren't that scary because if you fail them, what are you missing? If you pass, you get to write a big nasty dissertation--that's the really scary part. I kind of envy the people who didn't pass. I suppose I should have just quit, but I got too far before I realized I should, so I just had to grit my teeth and finish the stupid thing. Actually, I'm thinking I might be better off in terms of finding a job if I had learned more practical skills, rather than finishing. Should have just transferred to EE grad school is what I should have done.
I can't really say too many positive things about research because it's hard for me to imagine that anyone would like it that much, after my experience. I suppose it would have been cool if I didn't have to write it all down, which was extremely tedious. And if you took out the part where I messed up several times and had to throw out 5 pages of stuff and got set back a few months each time. And it would have helped if what I was doing had some relation to reality, other than waving my hands and saying, "hey, it could be relevant to string theory (but I don't even really know why exactly)".
I don't like teaching either, but I can say that if you do like teaching, that is a big advantage.
The other thing is that I may not even take any physics which in high school is what I always saw as being my calling. I had only had classical mechanics in my first year, but I like to read from textbooks from time to time and physics just doesn't grab me the way math does.
That sounds like a pretty premature conclusion. Having textbooks grab you is, in my experience, little indication that current research in pure math is going to grab you. It's a fundamentally different activity than studying from textbooks. If reading hundreds and hundreds of pages of very technical stuff that is very removed from reality is your thing, go for it. At some point, I just realized I needed to come back down to Earth. What I am really curious about is the way the world works, rather than playing abstract games, just for the sake of it, and that's why I realized I made a mistake to put so much time into studying pure math. Life is too short and there's too much to find out about the things that really matter. Math matters, too, but to me, it doesn't matter until it finds some application to science or if it leads you to deep philosophical conclusions.
Should I try continue to try it anyway (labs are what I really didn't like, is it advisable to just take non-lab courses or will grad schools think I'm just wasting their time?), is it possible to pick up physics in grad school if you wanted to do something math related?
I never liked the way labs are taught. Too cook-book. I want to do my own experiments, not follow someone else's procedure. That may be why you don't like it, and I wouldn't blame you.
Unless you pick just the right department/people to work with, they'll probably tend to sway you very far in the math direction. I tried to learn about physics on the side, but it was pretty hard to get very far with it, even though I studied topological quantum field theory, which is nominally related to physics.
-Switching from high prospects Engineering to Pure Math, am I crazy?
Yes, unless you really know what you are getting into and have thought it all through very carefully, which I would tend to doubt at this stage. Even I thought I had thought everything through, but it didn't work out as planned.
-Is it a waste of time to take physics without the lab component?
No. I understand a fair amount of physics with very little lab experience in physics. Granted, I would understand it better, probably, with lab experience, but I still understand it enough to make some use of it, mathematically.
-If I wanted to research something like mathematical physics in grad school is it actually necessary to have a physics background, or can you pick it up on the spot?
In my particular area, topological quantum field theory, you basically don't even need to know any physics at all (although it's a plus if you do know a lot). Literally, if you had never taken a single physics class, you could probably work in the field, although, there's a side of the literature that would be completely inaccessible to you. It's not really physics, though.