In simple cases like this, momentum of the overall system is conserved. In order to describe gravitational interactions accurately, we have to allow for the curvature of space, so we have to choose a coordinate system to map what is happening. The most practical coordinate system for the space around a single dominant central mass is an isotropic one, where the scale factor between each direction in local space and the corresponding coordinate in the coordinate space is the same, and the coordinate speed of light differs from the local speed of light only by a scalar factor (rather than a tensor).
According to Special Relativity, a photon of energy E and velocity \mathbf{v} has momentum \mathbf{p} = E \mathbf{v}/c^2 with magnitude E/c. If it is deflected overall by something through a small angle \alpha, expressed in radians, then the change in momentum has magnitude p \alpha perpendicular to the original path, so the change in momentum of the central mass is equal and in the opposite direction.
The same logic applies for gravity around a dominant central mass, described in isotropic coordinates. In that case, the total energy E remains a constant (as in Newtonian gravity, where the change in potential energy matches the change in kinetic energy), and the only slight difference is that c is now the coordinate speed of light, which varies slightly with potential, approximately as (1-2Gm/rc^2) times the standard value.
More generally, for weak fields described using isotropic coordinates, the following relativistic equation holds for all test objects, from bricks to photons, traveling in any direction, including radially inward or outward and tangentially, or anywhere in between:
\frac{d\mathbf{p}}{dt} = \frac{E}{c^2} \, \mathbf{g} \left ( 1 + \frac{v^2}{c^2} \right )
where \mathbf{g} is the effective Newtonian gravitational field. Since E is constant, we can also divide by it to get an equation of motion which does not depend on the energy of the test object:
\frac{d}{dt} \left ( \frac{\mathbf{v}}{c^2} \right ) = \frac{1}{c^2} \, \mathbf{g} \left ( 1 + \frac{v^2}{c^2} \right )
Note that even for a photon this still says that the downwards momentum is increasing at a rate which only depends on the field, regardless of its direction of travel. For example, if the photon travels downwards, then its momentum is E/c in the downwards direction, which increases with time because c decreases with potential.
The specific calculations change near a black hole, and there is a "photon" sphere radius (at Schwarzschild radial coordinate 3Gm/c^2) outside the black hole at which the curvature of the photon's path puts it into a circular (unstable) orbit. However, the general principle remains the same that (from the point of view of a suitable coordinate system) momentum is still conserved for the total system.