Autodidactic? Perhaps, but the same can be said for any subject as long as one doesn't have to take test. By chance [it's a long story] I never had a trigonometry class. I had to learn it on the fly while I was taking my first Calculus class!
It seems that I was extremely lucky. In the classical philosophy class, we mostly focused Aristotle, Plato, Descarte, and Socrates. It was a lot of work, very rewarding, and a life-changing experience. I also took a political philosophy class given by an old 60s radical - he [a white guy] actually rode on the bus with MLK through the South! For obvoius reasons, that class focused on modern political philosophies. It was interesting but not nearly as memorable as the classical stuff, from an academic point of view.
This thread makes me think there is a serious shortage of good philosophy teachers.
As for the difficulty compared to physics and math, well duh! Nothing can match the intellectual rigor of math and physics.