Let's define 'virtual particle' as internal lines in a Feynman diagram, i.e. lines with two ends at two vertices.
In a Feynman diagram energymomentum (and everything else like angular momentum, charge etc.) is conserved at every vertex; for virtual particles this means that they don't 'borrow energy from the vacuum'; all what they do is to violate the usual constraint pē  mē = 0 where p is the energemomentum four vector.
So for a virtual photon energymomentum conservation holds, but it represents something (attention: it's not a Fockstate!) with mē ≠ 0.
