The energy of the photon also determines the momentum of the photon. That is, a high energy photon is a high momentum photon.
In the Compton effect, an electron recoils after a collision with a photon, and this recoil acts just like a classical conservation of momentum collision.
A visible photon has much less energy than an x-ray. We're talking a thousandth to ten-thousandth less energy, and therefore that much less momentum.
A collision with a visible photn whould knock an electron into a higher shell, temporarily, but much more energy is required to knock it completely out of the material