We are following a geodesic in curved space time. A geodesic is an extension of the concept of a "straight line" to curved space time. And you can't leave time out of the equation.
It might help you understand by looking at a different non-Euclidean geometry, the geometry of the surface of the Earth. Ignoring the little bumps from mountains, dips from ocean trenches, is the equator a "straight line"? The equator is a circle. How could it possibly be a "straight line"?
The answer is that it is a straight line in the sense that a "straight line" between points A and B is the shortest of all possible paths between A and B. (Better said as "one of the shortest paths". The shortest path is not necessarily unique once you through out the parallel postulate). Consider the problem of going from point A to point B on the surface of the Earth. No tunneling is allowed. Each possible path from A and B must lie entirely on the surface of the Earth. In this sense, the equator is a "straight line" in the non-Euclidean geometry of a spherical surface.