I have looked at (3.17). I suppose that it is not enough to show that
$$2m'_{(a} \overline{m}'_{b)} -2k'_{(a} l'_{b)} = 2m_{(a} \overline{m}_{b)} -2k_{(a} l_{b)},$$
as this is obvious for transformation (3.17).
I will proceed in a pedestrian way, i.e., I will show that (3.17) defines a boost. Inverting (3.12) gives (using notation that I dislike)
$$\begin{align}
\boldsymbol{E}_4 &= \frac{1}{\sqrt{2}} \left( \boldsymbol{k} +\boldsymbol{l} \right) \\
\boldsymbol{E}_3 &= \frac{1}{\sqrt{2}} \left( \boldsymbol{k} - \boldsymbol{l} \right) .
\end{align}$$
Now define
$$\begin{align}
\boldsymbol{E}'_4 &= \frac{1}{\sqrt{2}} \left( \boldsymbol{k}' +\boldsymbol{l}' \right) \\
\boldsymbol{E}'_3 &= \frac{1}{\sqrt{2}} \left( \boldsymbol{k}' - \boldsymbol{l}' \right) ,
\end{align}$$
with ##\boldsymbol{k}'## and ##\boldsymbol{l}'## given by (3.17). Then, by (3.17),
$$\begin{align}
\boldsymbol{E}'_4 &= \frac{1}{\sqrt{2}} \left( A\boldsymbol{k} +A^{-1} \boldsymbol{l} \right) \\
&= \frac{1}{\sqrt{2}} \left[ \frac{A}{\sqrt{2}} \left( \boldsymbol{E}_4 + \boldsymbol{E_3} \right) + \frac{A^{-1}}{\sqrt{2}} \left( \boldsymbol{E}_4 - \boldsymbol{E_3} \right) \right] \\
&= \frac{1}{2} \left( A + A^{-1} \right) \boldsymbol{E}_4 +\frac{1}{2} \left( A - A^{-1} \right) \boldsymbol{E}_3
\end{align}$$
Since
$$\left[ \frac{1}{2} \left( A + A^{-1} \right) \right]^2 - \left[ \frac{1}{2} \left( A - A^{-1} \right) \right]^2 = 1, $$
we can set
$$\begin{align}
\cosh w &= \frac{1}{2} \left( A + A^{-1} \right) \\
\sinh w &= \frac{1}{2} \left( A - A^{-1} \right)
\end{align}$$
Something similar holds for ##\boldsymbol{E}_3##, so we have a boost with rapidity ##w## and speed ##v = \tanh w##.