When working with curved spaces, you have to generalize your definition of a straight line path. Specifically, in a curved space, light will travel along a "geodesic" which is the generalization of a straight line to curved space. A geodesic is the shortest path you can take between two points in a curved space, just like a straight line is the shortest path between two points in flat space.
So, You shouldn't think of the light traveling through curved space as if the light was traveling in flat space and turning around a curve. Instead, its the space itself that is bending.
Think about a light ray traveling in flat space. It travels the shortest path between two points, a straight line (the geodesic in flat space) and is not accelerating at all. Agree? Now the presence of the massive sun intrinsically warps the space itself, changing the geodesic the light will travel. But the light still travels along the geodesic, the shortest path.
A more familiar analogy may help: Think about walking in a "straight line" from Boston to Seattle. In reality you are walking along a curved path along the surface of the earth, but to you, stuck in the curved space of the Earth's surface, the path "feels" equivalent to a straight line. i.e. its the shortest path you can find.
