I don't think you need any exceptional talents to become good at mathematics. You just need a lot of interest (or patience + pain tolerance), time and practice. But you also have to be able to assess your skills correctly and fairly. If you loose your positive feedback mechanisms that motivate you, you are lost.
I too always had the impression that only the best were taken to PhD-schools but that's completely false. I always thought I wasn't good enough. But logics tell me there are obviously more PhD positions available than there are eligible "IQ 150+" people. Will you get anywhere without IQ 150? Sure, it will help, but not if you don't put a lot of time into your studies. I never studied for a maths exam before university, passed with highest grades in all subjects, got used to not working. Got into university, continued with my old tactics, passed all exams narrowly, which objectively still was good in relative terms as lots of people didn't pass them. However, ultimately, the gap between grades pre- and post-uni made me lose self esteem and finally interest, which made me fail exams, I didn't want to try anymore.
Add to that that the study environment at my uni was unfriendly, it seemed like many failed professors and PhDs just picked on the, initially enthusiastic, students. Students were forced to work day and night to survive the scientifically reasonable but perhaps somewhat sadistic and unfair requirements, and that of course didn't reflect well on the study environment. Other programmes at the uni had easier courses, competed with us, got higher grades easier and were able to go abroad, get well paid jobs, and other benefits because of that, which of course was unjust as they never normalized grades per programme.
I finally underrated my abilities, lost my interest to study (other than to pass my exams, I didn't care about studying the extra chapters for higher grades). It discouraged me from pursuing PhDs in physics and mathematics, even though I know I can manage it, and made me pursue an MD (where I at least have an unusual edge through the physics perspective). Society appreciates medical graduates more than physics graduates as well (even though we all know physics is the real science in this context), the environment is friendier, appreciative even, and I know I won't be unemployed, or wait 10 years for a full time contract, get fired or fooled around with. The backside is that I have tasted the fruit of physics, I know it's the real deal, while medicine, interesting as it is, unfortunately, is more about memory skills (and less hard evidence and deduction).
On the other hand, I know engineers and physicists (PhDs) who work as unskilled labor or in private elementary school classrom like sweat shops. They don't have fancy enough names to get finance jobs through their parents' contacts. Along with arts majors, teachers, talented artists and musicians, I think they are the ones who truly deserve the uttermost respect in society and not the Lamborghini economists, marketing people, venture capitalists, football players, or bubblegum pop singer stars. A lot of talent is put to waste, regardless of IQ or education. Don't let yourself or them waste yours.