We can use the chapter on spin from Grififths' Introduction to Quantum Mechanics to solve this problem. Define ##\vec{S}## as the total spin operator, with components ##S_x##, ##S_y## and ##S_z##. Also define the spin raising and lowering operators as ##S_\pm = S_x \pm i S_y\,##. For a state ##\ket{s,m}## we then have (in units of ##\hbar=1##):
$$
\begin{aligned}
S^2 \ket{s,m} = s(s+1)\ket{s,m}, \quad S_z \ket{s,m} = m\ket{s,m} \\[0.8em] \quad S_\pm \ket{s,m} = \sqrt{s(s+1)-m(m\pm1)} \ket{s,m\pm1}
\end{aligned} \tag 1$$To solve this problem, we will check that both the LHS and RHS have the same eigenvalue under ##S^2##. This will give us ##\alpha## and ##\beta## (up to an overall phase factor).
I'll introduce a little notation here. We will say that the RHS state is ##\ket\chi##, which is defined as
$$
\begin{aligned}
&\qquad\qquad\qquad\qquad\qquad \ket\chi = \alpha\ket{\chi_1} + \beta\ket{\chi_2}, \,\,\,\text{where} \\[0.8em]
&\ket{\chi_1} = \ket{s+1/2,m+1/2}\ket{1/2,-1/2}, \quad \ket{\chi_2} = \ket{s+1/2,m-1/2}\ket{1/2,1/2}
\end{aligned} \tag 2$$
The operator ##S^2## which now acts on both spin states of the RHS can be written as
$$S^2 = (S^{(1)})^2 + (S^{(2)})^2 + 2 \vec{S}^{(1)} \cdot \vec{S}^{(2)}$$ where the components ##S^{(1)}## and ##S^{(2)}## act on the spin-##s## and spin-##1/2## components of ##\ket\chi## respectively.
It will be helpful to express the product ##2 \vec{S}^{(1)} \cdot \vec{S}^{(2)}## in terms of the raising and lowering operators. With a little algebra, we have
$$
\begin{aligned}
2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} &= 2 \left( S_x^{(1)}S_x^{(2)} + S_y^{(1)}S_y^{(2)} + S_z^{(1)}S_z^{(2)} \right) \\
&= \ldots \\
&= S_+^{(1)}S_-^{(2)} + S_-^{(1)}S_+^{(2)} + 2S_z^{(1)}S_z^{(2)}
\end{aligned}
\tag 3
$$Finally, we are ready to compute the effect of ##S^2## on ##\ket\chi##. Using equations (1), (2) and (3) we have
$$
\begin{aligned}
2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} \ket{\chi_1} &= \sqrt{(s+1)^2-m^2} \ket{\chi_2} - (m+1/2)\ket{\chi_1} \\
2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} \ket{\chi_2} &= \sqrt{(s+1)^2-m^2} \ket{\chi_1} + (m-1/2)\ket{\chi_2}
\end{aligned}
\tag 4
$$It is easy to see that ##\ket\chi## is an eigenstate of ##(S^{(1)})^2## and ##(S^{(2)})^2##, and we expect it to be an eigenstate of ##S^2## as well. Hence, it should be an eigenstate of ##\vec{S}^{(1)} \cdot \vec{S}^{(2)}##. Observe that using (4), we can write
$$
2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} \ket\chi = \alpha \left( -(m+1/2) + \frac \beta \alpha \sqrt{(\ldots)} \right) \ket{\chi_1} + \beta \left( (m-1/2) +\frac \alpha \beta \sqrt{(\ldots)} \right) \ket{\chi_2} \tag 5
$$Hence for ##\ket\chi## to be an eigenstate, we must have
$$
-(m+1/2) + \frac \beta \alpha \sqrt{(\ldots)} = (m-1/2) +\frac \alpha \beta \sqrt{(\ldots)} \tag 6
$$Solving for ##\beta/\alpha## yields
$$
\frac \beta \alpha = \frac{m \pm (s+1)}{\sqrt{(s+1)^2-m^2}} \tag 7
$$Substituting back into (5), we see that
$$
2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} \ket\chi = (\pm s \pm 1 - 1/2) \ket\chi \tag 8
$$Computing the rest of ##S^2 \ket\chi## is simple:
$$
\begin{aligned}
S^2 \ket\chi &= (S^{(1)})^2 \ket\chi + (S^{(2)})^2 \ket\chi + 2 \vec{S}^{(1)} \cdot \vec{S}^{(2)} \ket\chi \\[0.8em]
&= (s+1/2)(s+3/2) \ket\chi + (1/2)(3/2) \ket\chi + (\pm s \pm 1 - 1/2) \ket\chi
\end{aligned}
\tag 9$$We want the negative case because it yields the eigenvalue ##s(s+1)##, which matches the eigenvalue when ##S^2## acts on ##\ket{s,m}##. Hence, we take the negative case of (7)
$$
\frac \beta \alpha = \frac{m-(s+1)}{\sqrt{ (s+1)^2 - m^2 }}
$$We can then use normalisation to obtain the coefficients ##\alpha## and ##\beta##. Because ##\ket{\chi_1}## and ##\ket{\chi_2}## are orthonormal (and that ##\ket{s,m}## is correctly normalised), we must have ##\alpha^2 + \beta^2 = 1##. After some more algebra, we obtain
$$
\alpha = \pm \sqrt{\frac{s+m+1}{2s+2}}, \quad \beta = \mp \sqrt{\frac{s-m+1}{2s+2}}
$$which matches the expression in the paper.
