I think people tend to be dreadful at mathematics because it is not taught properly. Calculus isn't just useful for analyzing physics and chemistry and assorted other topics, it has explanatory value in and of itself. Again speaking from personal experience, algebra became so much clearer once I knew why
those techniques were taught. For instance, I couldn't give a darn less about multiplying by conjugates until I started studying limits. I didn't appreciate the beauty of e until I heard the definition of the natural logarithm. I certainly didn't much understand the purpose of being able to discern the equation of a line from a point and a slope. To me, learning calculus is like a myopic person putting on glasses. Sure, you can vaguely understand some concepts, but it's so much clearer
in light of calculus.