I think this view is mainly created by the media and popular culture from 20th century up until now because of the giant leap in scientific discovery and technological advancements of the time. A subject like physics will sound challenging because people will immediately connect that to Albert Einstein, relativity, moon landing, atomic bombs and etc. Those are certainly something that is 'difficult', but when you actually learn those subject you don't get exposed to that directly, instead you will learn step by step from the simplest concept, like distance is speed multiplied by time. Something that I think everyone knows.
Most people are in my opinion generally proficient in basic math and arithmetic without quite realizing it, because our everyday lives require it i.e you need to be able to calculate change, determine how much tips you want to pay and so on. Of course I don't deny that at times they can be tougher than other subjects mainly due to the abstraction and the unintuitive aspects in it. Say Newton first law is already unintuitive because you really don't encounter such behavior everyday. If you stop pushing your supermarket trolley, well it stops.
For me personally singing is very challenging, not only because I don't have much knowledge in musical theory, but also because of the embarrassment I have to endure. Drawing can be quite difficult if you are not trained well. My friend who likes drawing manga even needs to learn human anatomy extensively and more in the college.
There are also other subjects that are no less challenging than science and math; say languages for example. You obviously speak everyday, but can you master 6-5 languages fluently, know the intricacies of their grammars, idiom, different dialects and etc? There are people who devoted their time just to do that in linguistic study.