The relative velocity for two particles, where at least one particle (say particle 1) has non-zero mass is the velocity of the other particle in the rest frame of particle 1 (which is called the lab frame in scattering theory for historical reasons, when the accelerator experiments where fixed-target experiments). It's easy to formulate this in a manifestly covariant way. In the lab frame we have for the four-momenta (I use natural units, ##c=1##)
$$p_1^*=(m_1,0)^T, \quad p_2^*=(E_2^*,\vec{p}_2^*) \; \Rightarrow \; v_{\text{rel}}=\frac{|\vec{p}_2^*|}{E_2^*}=\sqrt{E_2^{*2}-m_2^2}{E_2^*}.$$
Now we have for the Minkowski product in this frame and thus for any other frame
$$p_1^* \cdot p_2^*=p_1 \cdot p_2=m_1 E_2^* \; \Rightarrow \; E_2^*=\frac{p_1 \cdot p_2}{m_1},$$
and thus
$$E_2^{*2}-m_2^2=\frac{(p_1 \cdot p_2)^2}{m_1^2}-m_2^2=\frac{(p_1 \cdot p_2)^2-m_1^2 m_2^2}{m_1^2},$$
which leads to
$$v_{\text{rel}}=\frac{\sqrt{(p_1 \cdot p_2)^2-m_1^2 m_2^2}}{p_1 \cdot p_2}.$$
Now, if ##m_1=m_2=0## you have two possibilities: ##p_1 \cdot p_2=0##; then the relative velocity is undefined because then ##p_1 \propto p_2## or ##p_1 \cdot p_2 \neq 0##, and the relative velocity is 1. If ##m_1 \neq 0## but ##m_2=0## you have always ##v_{\text{rel}}=1##.
Also note that ##v_{\text{rel}}## is only ##|\vec{v}_1-\vec{v}_2|## if ##\vec{v}_1 \propto \vec{v}_2##. The general expression in an arbitrary frame is
$$v_{\text{rel}}=\frac{1}{1-\vec{v}_1 \cdot \vec{v}_2} \sqrt{(\vec{v}_1-\vec{v}_2)^2-(\vec{v}_1 \times \vec{v}_2)^2}.$$
For a derivation, see
http://th.physik.uni-frankfurt.de/~hees/pf-faq/srt.pdf
Sect. 1.5 and 1.6.