Imagine that two electrons interact by exchanging a virtual photon. Electron A gains momentum ##-\vec{p}## and electron B gains momentum ##\vec{p}##. If the two momentum vectors are not collinear then there will be extra angular momentum left over from the interaction. In a simple Coulomb interaction the momenta of A and B are collinear but I would have thought that in a general interaction they would not be. In that case how would angular momentum be conserved? PS As the photon is virtual there isn't anything left in the EM field.
I can only think in classical terms using the Lienard-Wiechert theory. If electron A has an acceleration perpendicular to the line A-B then electron B will receive some momentum perpendicular to A-B opposite A's acceleration. Therefore the total momentum, ##\vec{p}##, transferred to B will not be parallel to A-B.