Use the square of the distance formula.
Also, and this may be what you are looking for, instead of using xy2z3= 2 to replace one variable with the other two, use "Lagrange multipliers". If we write f(x,y,z)= x2+ y2+ z2, the square of the distance to the origin, and g(x,y,z)= xy2z3= 2, then max or min values of f, for points that satisfy g(x,y,z)= 2 must have [itex]\nabla f[/itex] parallel to [itex]\nabla g[/itex]- one must be a multiple of the other. Setting [itex]\nabla f= \lambda g[/itex] and comparing the components, together with g(x,y,z)= 2, gives 4 equations to solve for x, y, z, and [itex]\lambda[/itex].
Tip: since you are not interested in the value of [itex]\lambda[/itex], and [itex]\lambda[/itex] is simply multiplied by the functions of x, y, and z, often a best first thing to do is to divide one equation by another to eliminate [itex]\lambda[/itex].