I don't think you need to be a genius but I think that only 1 to 10% of people have the reasoning skills to critically relate together the information they digest. I am amazed the amount of people that can't follow non mathamatical discussions in phillosophy and economics. When it comes to math even basic calculus and statistics is lost on many. Many people whove studied calculus lose even the basic conceptial understanding of what a derivative is after school and many people who have studied basic economics have forgoten basic supply and demand.
Simple elementry concepts make many peoples head hurt. People whose heads were full of ideas often lose the energy to think they once had when they age. Sure, with effort people could reach and maintain a higher level of cognition but not everyone has the same mental energy. The human mind is both limited by memory, the rate we can digest various kinds of information and the amount of information we are able to relate together. An average person can only hold about 5 pieces of information in their short term memory at once so in order to think more broadly one needs to think with their subcontios or with something external to themselves like paper or a computer.
However, with all the above said perhaps I am biassed on what Ideas and concepts people should retain.