Let's define 'virtual particle' as internal lines in a Feynman diagram, i.e. lines with two ends at two vertices.
In a Feynman diagram energy-momentum (and everything else like angular momentum, charge etc.) is conserved at every vertex; for virtual particles this means that they don't 'borrow energy from the vacuum'; all what they do is to violate the usual constraint pē - mē = 0 where p is the energe-momentum four vector.
So for a virtual photon energy-momentum conservation holds, but it represents something (attention: it's not a Fock-state!) with mē ≠ 0.