Here's where Chestermiller's suggestion comes from. The idea is to find R and φ such that
\begin{align*}
A &= R \cos \varphi \\
B &= R \sin \varphi
\end{align*} so that
\begin{align*}
A \sin\theta - B \cos \theta &= (R \cos\varphi)\sin\theta - (R\sin\varphi)\cos\theta \\
&= R(\cos\varphi\sin\theta - \sin\varphi\cos\theta) \\
&= R \sin(\theta-\varphi).
\end{align*} If you divide the second equation by the first, you get
$$\frac{B}{A} = \frac{R\sin\varphi}{R\cos\varphi} = \tan\varphi,$$ and using the Pythagorean identity, you get
$$1 = \cos^2 \varphi + \sin^2\varphi = \left(\frac{A}{R}\right)^2 + \left(\frac{B}{R}\right)^2,$$ or equivalently, ##R^2 = A^2+B^2##.
This trick is a good one to know. As he noted, it's a lot less tedious to solve the problem using this method than dealing with the quadratic equation, especially when you have non-integer numbers like the ones in this problem.