Hello PolkaDots 54,
We'll use the Spectral theorem. We can express $3x^2 + 2xy + 3y^2 - 8 = 0$:
$$(x,y) \begin{pmatrix}{3}&{1}\\{1}&{3} \end{pmatrix} \begin{pmatrix}{x}\\{y}\end{pmatrix}=8 \Leftrightarrow (x,y)A \begin{pmatrix}{x}\\{y}\end{pmatrix}=8$$ Eigenvalues of $A$:
$$\chi(\lambda)=\det (A-\lambda I)=\begin{vmatrix}{3-\lambda}&{1}\\{1}&{3-\lambda}\end{vmatrix}=\lambda^2-6\lambda+8=0\Leftrightarrow \lambda= 2\:\vee\;\lambda=4$$ Basis of the eigenspaces: $$\ker\;(A-2I)\;\equiv\left \{
\begin{array}{rcrcr}
\,x_1 & + & \,x_2 & = & 0 \\
\,x_1 & + & \,x_2 & = & 0
\end{array}
\right .\quad ,\quad B_2=\{(1,-1)\}$$
$$\ker\;(A-4I)\;\equiv\left \{
\begin{array}{rcrcr}
-\,x_1 & + & \,x_2 & = & 0 \\
\,x_1 & - & \,x_2 & = & 0
\end{array}
\right .\quad ,\quad B_4=\{(1,1)\}$$ Eigenvectors orthonormal basis of $\mathbb{R}^2$:
$$B=\left\{\frac{1}{\sqrt{2}}(1,-1),\frac{1}{\sqrt{2}}(1,1)\right\}$$ Change of basis matrix (orthogonal because its columns are orthonormal) :
$$P=\frac{1}{\sqrt{2}}\begin{pmatrix}{\;\;1}&{1}\\{-1}&{1}\end{pmatrix}$$ Change of coordinates:
$$\begin{pmatrix}{x}\\{y} \end{pmatrix}=P \begin{pmatrix}{x'}\\{y'} \end{pmatrix}$$ Then, $$(x,y)A \begin{pmatrix}{x}\\{y} \end{pmatrix}=8\Leftrightarrow (x',y')P^TAP \begin{pmatrix}{x'}\\{y'} \end{pmatrix}=8\Leftrightarrow (x',y')P^{-1}AP \begin{pmatrix}{x'}\\{y'} \end{pmatrix}=8\\ (x',y') \begin{pmatrix}{2}&{0}\\{0}&{4} \end{pmatrix} \begin{pmatrix}{x'}\\{y'} \end{pmatrix}=8\Leftrightarrow 2(x')^2+4(y')^2=8\Leftrightarrow (x')^2+2(y')^2=4$$ The equation of the conic with respect to $B$ is $(x')^2+2(y')^2=4$ or equivalently $$\boxed{\dfrac{(x')^2}{2^2}+\dfrac{(y')^2}{(\sqrt{2})^2}=1\quad (\mbox{ellipse})}$$
