The hard part is to figure out how to teach QM without a lot of math. I think that relativity can be taught without a lot of math, but I don't see how QM can be. The problem is that QM is weird. If you wanted to teach Newtonian mechanics without math, you can do it by analogy. There is a ball, throw the ball, watch the ball. You can't do that with QM, and efforts to teach QM by analogy end up being more confusing than helpful.
That was the attitude of US educators until the 1950's when French and Taylor put together a course for undergraduates on QM at MIT. It takes a math education genius to make the math easy. A lot of the pressure was Cold War. If the US doesn't teach QM then we'll be waving red flags when the Russians do, and essentially the President and the generals ordered MIT to figure out how to teach QM quickly.
Curiously the Russians were under similar pressures, and I've seen excellent educational material translated from Russian, and part of me is sad that the Cold War ended with a total US victory rather than a "friendly stalemate."
A lot of the new techniques of physics education got started at MIT (undergraduate research and the trend away from lecture courses toward small interactive groups).
MIT is pretty innovative as far as physics education goes because all MIT students have to take basic physics, and having a group of 1000 math-literate students gives you a lot of lab rats for new stuff.
One fun thing is to go back to the mid-19th century and read William Barton Rogers. There is one essay in which he explains step by step the idea that the student will learn material from an instructor, and then be giving a written test at which he will be given a numerical score for that test. It looks silly, because it's so obvious, but then you realize that it's obvious today because everyone does it, and he was trying to explain the new system to people that had never seen it before.