Use the square of the distance formula.
Also, and this may be what you are looking for, instead of using xy2z3= 2 to replace one variable with the other two, use "Lagrange multipliers". If we write f(x,y,z)= x2+ y2+ z2, the square of the distance to the origin, and g(x,y,z)= xy2z3= 2, then max or min values of f, for points that satisfy g(x,y,z)= 2 must have \nabla f parallel to \nabla g- one must be a multiple of the other. Setting \nabla f= \lambda g and comparing the components, together with g(x,y,z)= 2, gives 4 equations to solve for x, y, z, and \lambda.
Tip: since you are not interested in the value of \lambda, and \lambda is simply multiplied by the functions of x, y, and z, often a best first thing to do is to divide one equation by another to eliminate \lambda.