Hello Emma,
We are given to solve:
$$\cos^2(x)-8\sin(x)\cos(x)+3=0$$ where $$0^{\circ}\le x\le360^{\circ}$$
If we use the following double-angle identities for sine and cosine:
$$\cos(2\theta)=2\cos^2(\theta)-1\,\therefore\,cos^2(\theta)=\frac{1+\cos(2\theta)}{2}$$
$$\sin(2\theta)=2\sin(\theta)\cos(\theta)$$
then the equation becomes:
$$\frac{1+\cos(2x)}{2}-4\sin(2x)+3=0$$
which we can arrange as:
$$8\sin(2x)-\cos(x)=7$$
Now, if we define (where $$k\in\mathbb{R},\,0^{\circ}<\alpha<90^{\circ}$$):
$$8=k\cos(\alpha)$$
$$1=k\sin(\alpha)$$
then we obtain by division:
$$\tan(\alpha)=\frac{1}{8}\,\therefore\,\alpha=\tan^{-1}\left(\frac{1}{8} \right)$$
and by squaring and adding:
$$8^2+1^2=65=k^2\left(cos^2(\alpha)+\sin^2(\alpha) \right)=k^2\,\therefore\,k=\sqrt{65}$$
Hence, our equation becomes:
$$\cos(\alpha)\sin(2x)-\sin(\alpha)\cos(x)=\frac{7}{k}$$
Using the angle-difference identity for sine, and substituting for $\alpha$ and $k$ we obtain:
$$\sin\left(2x-\tan^{-1}\left(\frac{1}{8} \right) \right)=\frac{7}{\sqrt{65}}$$
Because of the periodicity of the sine function, we may write, where $$k\in\mathbb{Z}$$
(1) $$\sin\left(2x-\tan^{-1}\left(\frac{1}{8} \right)+k\cdot360^{\circ} \right)=\frac{7}{\sqrt{65}}$$
Now, combining this with the identity $$\sin\left(180^{\circ}-\theta \right)=\sin(\theta)$$ we also have:
(2) $$\sin\left(\tan^{-1}\left(\frac{1}{8} \right)-2x+(2k+1)\cdot180^{\circ} \right)=\frac{7}{\sqrt{65}}$$
Taking the inverse sine of both sides of (1), we find:
$$2x-\tan^{-1}\left(\frac{1}{8} \right)+k\cdot360^{\circ}=\sin^{-1}\left(\frac{7}{\sqrt{65}} \right)$$
Solving for $x$, we find:
$$x=\frac{1}{2}\left(\sin^{-1}\left(\frac{7}{\sqrt{65}} \right)+\tan^{-1}\left(\frac{1}{8} \right) \right)-k\cdot180^{\circ}$$
Thus, for appropriate values of $k$, we find the solutions on the given interval for $x$ of:
$$k=0\implies x=\frac{1}{2}\left(\sin^{-1}\left(\frac{7}{\sqrt{65}} \right)+\tan^{-1}\left(\frac{1}{8} \right) \right) \approx33.69006752598^{\circ}$$
$$k=-1\implies x=\frac{1}{2}\left(\sin^{-1}\left(\frac{7}{\sqrt{65}} \right)+\tan^{-1}\left(\frac{1}{8} \right) \right)+180^{\circ} \approx213.69006752597983^{\circ}$$
Taking the inverse sine of both sides of (2), we find:
$$\tan^{-1}\left(\frac{1}{8} \right)-2x+(2k+1)\cdot180^{\circ}=\sin^{-1}\left(\frac{7}{\sqrt{65}} \right)$$
Solving for $x$, we find:
$$x=\frac{1}{2}\left(\tan^{-1}\left(\frac{1}{8} \right)-\sin^{-1}\left(\frac{7}{\sqrt{65}} \right) \right)+(2k+1)\cdot90^{\circ}$$
Thus, for appropriate values of $k$, we find the solutions on the given interval for $x$ of:
$$k=0\implies x=\frac{1}{2}\left(\tan^{-1}\left(\frac{1}{8} \right)-\sin^{-1}\left(\frac{7}{\sqrt{65}} \right) \right)+90^{\circ} \approx63.43494882292201^{\circ}$$
$$k=1\implies x=\frac{1}{2}\left(\tan^{-1}\left(\frac{1}{8} \right)-\sin^{-1}\left(\frac{7}{\sqrt{65}} \right) \right)+270^{\circ} \approx243.43494882292202^{\circ}$$
