This is one of those cases where feynman diagrams give a misleading picture.
It's easy to imagine an exchange of a photon generating a repulsive force due to conservation of momentum, but virtual photons are different than regular photons. They can have qualities we don't associate with regular photons, which muddies up the issue.
Also, virtual photons might not actually exist, but they do make a useful device in calculations. By their very nature, all detected photons cannot be virtual photons.