There are, in my opinion, two reasons why people do not like math
1) Math is difficult to grasp. I have really struggled through the first year of math in elementary school, I found it quite difficult. But I persisted, and suddenly it all clicked. And suddenly, I found it all easy and I liked it. However, most people don't have the patience to persist and to wait until it clicks. It also requires so maturity, which most people don't have at that age.
Furthermore, it takes a little while to get to the interesting stuff. You need to do boring addition and multiplication first, solving linear systems, solving quadratic equations,... When the math finally gets interesting, many people have given up.