Use "Lagrange multipliers". Since [itex]w(x,y)= (x- 1)^2+ (y- 1)^2[/itex], [itex]\nabla w= 2(x- 1)\vec{i}+ 2(y- 1)\vec{j}[/itex]. Writing the constraint as [itex]f(x,y)= px^3+ qx^2y+ rxy^2+ ty^3[/itex], [itex]\nabla f= (3px^2+ 2qxy+ ry^2)\vec{i}+ (qx^2+ 3rxy+ 3ty^3)\vec{j}[/itex]. At a max or min of w, with constraint f, those two gradient vectors must be parallel. That is, there must exist a number, [itex]\lambda[/itex] (the "Lagrange Multiplier") such that [itex]2(x- 1)\vec{i}+ 2(y- 1)\vec{j}[/itex][itex]= \lambda[(3px^2+ 2qxy+ ry^2)\vec{i}+ (qx^2+ 3rxy+ 3ty^3)]\vec{j}[/itex].
That gives the equations [itex]2(x- 1)= \lambda (3px^2+ 2qxy+ ry^2)[/itex] and [itex]2(y- 1)= \lambda(qx^2+ 3rxy+ 3y^3)[/itex] which, together with the constraint, give three equations to solve or x, y, and [itex]\lambda[/itex]. Since a value for [itex]\lambda[/itex] is not part of the solution to this problem, I find that it is often simplest to first eliminate [itex]\lambda[/itex] by dividing one equation by the other. Dividing the first of those two equations by the second,
[tex]\frac{2(x- 1)}{2(y- 1}= \frac{3px^2+ 2qxy+ ry^2}{qx^2+ 2rxy+ 3ty^3}[/tex].
That is, of course, equivalent to [tex](x- 1)(qx^2+ 2rxy+ 3ty^3)= (y- 1)(3px^2+ 2qxy+ ry^2)[/tex].