Karol said:
$$2x^2+8y^2=4~\rightarrow~y^2=\frac{1}{2}-\frac{1}{4}x^2$$
$$s^2=x^2+y^2=x^2+\frac{1}{2}-\frac{1}{4}x^2=\frac{3}{4}x^2+\frac{1}{2}$$
$$s^2_{min}(x=0)=\frac{1}{4}~\rightarrow~s_{min}=\frac{1}{2}$$$$5(x^2+y^2)-6xy=4~\rightarrow~xy=\frac{5}{6}(x^2+y^2)-\frac{2}{3}$$
$$\gamma (x^2 + y^2)=\gamma s^2 \geq (x^2 y^2 )^{\frac{1}{2}} = xy=\frac{5}{6}(x^2+y^2)-\frac{2}{3}$$
$$\gamma s^2 \leq \frac{12}{5-6\gamma}$$
It doesn't lead to anything.
I tried an other thing:
$$xy=\sqrt{x^2y^2}=\sqrt{ \left[ \frac{5}{6}(x^2+y^2)-\frac{2}{3} \right ]}^2=\left[ \frac{25}{36}(x^2+y^2)^2-\frac{20}{18}(x^2+y^2)+\frac{4}{9} \right]^{1/2} \leq \gamma(x^2+y^2)$$
No where
##GM \leq AM## seems to be one of my favorite inequalities these days, so what I was thinking of was actually was
Karol said:
$$5x^2-6xy+5y^2=4~\rightarrow~x^2+y^2=\frac{4+6xy}{5}$$
your goal: minimize ##x^2+y^2 = LHS##. This is an equality, so the Left Hand Side (LHS) is minimized if and only if the Right Hand Side (RHS) is.
##RHS = \frac{4+6xy}{5} = \frac{4}{5} + \frac{6}{5}xy = constant + \frac{6}{5}xy##
an additive constant is not directly important in optimization problems (another way to think about why they get mapped to zero by the derivative operator), so the RHS is minimized if and only if ##\frac{6}{5}xy## is minimized which is minimized if and only if ##xy## is minimized.
Draw the picture and split cases into two.
Top right and bottom left quadrants of your graph
##RHS = constant + \frac{6}{5}\big \vert xy \big \vert##
the scaling is positive, so ##\frac{6}{5}\big \vert xy \big \vert \geq 0##
Top left and bottom right quadrants
##RHS = constant + \frac{3}{5}\Big(-1*\big \vert 2 xy \big \vert\Big)##
it's immediately apparent we want top left and bottom right quadrants, as we can scale the magnitude of ##\big \vert 2 xy \big \vert## by a negative number, and hence shrink the RHS.
Goal: maximize ##\big \vert 2 xy \big \vert##. Apply ##GM \leq AM##
##\big \vert 2 xy \big \vert = 2\big \vert x \big \vert \big \vert y \big \vert \leq \big \vert x\big \vert^2 + \big \vert y\big \vert^2 = x^2 + y^2##
with equality if and only if
## \big \vert x \big \vert= \big \vert y \big \vert ##
and since we are in top left and bottom right quadrants, this means
##x = -y##
- - - -
edit: to be extra clear, the above tells us that for minimization we're dealing in top left or bottom right quadrant, and we have
##x^2 + y^2 = constant + \frac{3}{5}\Big(-1*\big \vert 2 xy \big \vert\Big) \geq constant - \frac{3}{5} (x^2 + y^2)##
add ## \frac{3}{5} (x^2 + y^2)## to each side
thus we have
##\frac{8}{5}\big(x^2 + y^2\big) \geq constant##
giving you the minimum value
##\big(x^2 + y^2\big) \geq \frac{5}{8}constant##
and from above, we know reaching this minimum occurs if and only if ##x = -y##
- - - -
To be real clear, my approach is start by sketching the graph and looking at it. Then I look at the symbols involved -- in this case I see products on RHS and sums on LHS and I look for a way to go from products to sums. ##GM \leq AM## is one way to do this.