[itex]x^2+ xy- 2y^2= 0[/itex]
[itex]x^2+ xy+ (1/4)y^2- (1/4)y^2- 2y^2= 0[/itex]
[itex](x+ (1/2)y)^2- (9/4)y^2= 0[/itex]
That's of the form "[itex]a^2- b^2= (a- b)(a+ b)[/itex]with a= x+ (1/2)y and b= (3/2)y
so [itex]x^2+ xy- 2y^2= (x+ (1/2)y+ (3/2)y)(x+ (1/2)y- (3/2)y)= (x+ 2y)(x- y)[/itex]
That's one way to do it- very "algorithmic". Another way, more "heuristic", is to note that the only way to factor [itex]x^2[/itex] (with integer coefficients) is (x)(x) while there are two ways to factor [itex]-2y^2[/itex] (again with integer coefficients) is (-2y)(y) or (2y)(-y). We try (-2y)(y): [itex](x- 2y)(x+ y)= x^2- 2xy+ xy- 2y^2= x^2- xy- 2y^2[/itex]. No, that's not what we want. So try (-y)(2y): [itex](x+ 2y)(x- y)= x^2+ 2xy- xy- 2y^2= x^2+ xy- 2y^2[/itex]. Yes, that's what we want.
[itex]6x^2+ 5xy- 6y^2= 6(x^2+ (5/6)xy- y^2)= 6(x^2+ (5/6)xy+ (25/144)y^2- (25/144)y^2- y^2)= 6((x+ 5/12)^2- (169/144)y^2)= 6((x+ 5/12)^2- ((13/12)y)^2)= 6(x+ 5/12+ 13/12)(x+ 5/12- 13/12))= 6(x+ 18/12)(x- 8/12)= 6(x+ 3/2)(x- 2/3)= (2x+ 3)(3x- 2)[/itex].
Or: 6=(3)(2) so we must have one of (3x- 3y)(2x+ 2y), (2x- 2y)(3x+ 3y), (3x- 2y)(2x+ 3y), or (2x- 3y)(3x+ 2y). Multiplying each of those out, we see that
[itex](3x- 3y)(2x+ 2y)= 6x^2+ 6xy- 6xy- 6y^2= 6x^2- 6y^2[/itex]
[itex](2x- 2y)(3x+ 3y)= 6x^2+ 6xy- 6xy- 6y^2= 6x^2- 6y^2[/itex]
[itex](3x- 2y)(2x+ 3y)= 6x^2+ 9xy- 4xy- 6y^2= 6x^2+ 5xy- 6y^2[/itex]
[itex](2x- 3y)(3x+ 2y)= 6x^2+ 4xy- 9xy- 6y^2= 6x^2- 5xy- 6y^3[/itex]
so that (3x- 2y)(2x+ 3y) is the correct factoring.
And we see that