But look at the units... The acceleration is measured in meters per second per second
, meaning that every second the speed increases by 9.8 meters/sec.... Let that go for a few hundred seconds and you've built up some serious speed. (note: gravitational acceleration is only 9.8 m/sec at the surface of the earth; it declines steadily the further away from earth you are).
It turns out that an object falling from infinity to the surface of the earth will reach a speed of about 11 km/sec; that's its gravitational potential energy being converted into kinetic energy. It's a lot.