Ok, let's have particles with three-velocities ##\vec{v}_1## and ##\vec{v}_2## wrt. to an inertial reference frame. The first step is to translate this to the appropriate covariant objects. In this case that's the four-velocities of the (on-shell) particles which are the four-velocities
$$u_j^{\mu}=\gamma_j (1,\vec{v}_j), \quad \gamma_j=\frac{1}{\sqrt{1-\vec{v}_j^2}}, \quad c=1, \quad j \in \{1,2 \}$$
They are related to the four-momenta of the particles by
$$p_j^{\mu}=m_j u_j^{\mu},$$
where ##m_j## are the invariant (or rest) masses of the particles, but this we don't need.
The relative velocity is now defined as the velocity of particle 2 in the rest frame of particle 1. Thanks to the covariant tensor formalism you don't need to perform the Lorentz transform here (although that's a nice exercise, you should do for yourself, but it's also a bit tedious to type here in the forum). The trick is to express everything covariantly. In the rest frame of particle 1 (in the following written with a tilde over the components of four-vectors) you have
##\tilde{u}_1^{\mu}=(1,0,0,0), \quad \tilde{u}_2^{\mu} = \tilde{\gamma}_2 (1,\vec{\tilde{v}_2}).##
From this it's clear that
##\tilde{\gamma}_2=u_2^0=\tilde{u}_{1,\mu} \tilde{u}_2^{\mu}.##
But now, this is a scalar expression (the Minkowski product of two four-vectors) and thus you have
##\tilde{\gamma}_2=u_1 \cdot u_2=\gamma_1 \gamma_2 (1-\vec{v}_1 \cdot \vec{v}_2),##
which is the formula you wanted to derive in posting #1.
One should keep in mind that the Lorentz transform is at the heart of deriving the Minkowski space as the space-time model, establishing the four-tensor formalism. For practical calculations you usually don't need Lorentz transforms, if everything is formulated in covariant form, and you can express the quantities you want to know in terms of covariant equations, which is the case for all physically interesting quantities. Sometimes one has to define appropriate quantities in a covariant way (e.g., temperature and other thermodynamic quantities and material constant, where usually you define them in the (local) rest frame of the matter and then express this definition in a manifestly covariant way or the invariant cross sections for processes in high-energy physics).
