[sp]This is just a variation on Mark's solution, from a geometric viewpoint (so as to avoid Lagrange multipliers at all costs (Tongueout)).
Write the equation as $\sqrt{2a^2 + 2ab + 3b^2} = 4 - a - b$ and square both sides, to get $2a^2 + 2ab + 3b^2 = 16 - 8a - 8b + a^2 + b^2 + 2ab$, which simplifies to $$a^2 + 2b^2 + 8a + 8b = 16.$$ I prefer to use $x$ and $y$ instead of $a$ and $b$, so this becomes $$x^2 + 2y^2 + 8x + 8y = 16.$$ (This is the equation of an ellipse centred at $(-4,-2)$.) We want to find the maximum value of $x^2y$ at a point on this ellipse. So we need to find the largest value of $k$ for which the curve $x^2y=k$ intersects the ellipse somewhere in the positive quadrant. That will happen when the two curves just touch at a single point. In that case, the two curves will share a common tangent at that point, and in particular they will have the same slope at that point. So, differentiate the equations of the curves to get $$2x + 4yy' + 8 + 8y' = 0,\qquad 2xy + x^2y' = 0.$$ If those equations both hold at the same point $(x,y)$ then $$y' = -\frac{x+4}{2(y+2)} = -\frac{2y}x,$$ so that $x(x+4) = 4y(y+2)$. Write that as $x^2 - 4y^2 + 4(x-2y) = 0.$ Factorise the difference of two squares, getting $(x-2y)(x+2y+4) = 0.$
Now, just as in Mark's solution, $x+2y+4$ cannot be zero, so we must have $x=2y.$ Substitute that into the ellipse equation, getting $6y^2 + 24y - 16 = 0$, or $3y^2 + 12y -8=0.$ Solve that quadratic, to get the positive solution $y = -2 + \frac23\sqrt{15}.$
From the quadratic, $y^2 = -4y + \frac83$,so $$y^3 = -4y^2 + \tfrac83y = -4\bigl(-4y + \tfrac83\bigr) + \tfrac83y = \tfrac{56}3y - \tfrac{32}3 = \tfrac{56}3\bigl(-2 + \tfrac23\sqrt{15}\bigr) - \tfrac{32}3 = -48 + \tfrac{112}9\sqrt{15}.$$
Finally, $$x^2y = (2y)^2y = 4y^3 = -192 + \tfrac{448}9\sqrt{15} = 64\bigl(-3 + \tfrac79\sqrt{15}\bigr).$$ That represents the maximum value of $x^2y$ on the ellipse. Mercifully, this agrees with Mark's result.
In the graph below, the orange ellipse is $x^2 + 2y^2 + 8x + 8x = 16,$ the blue curve is $x^2y = 64\bigl(-3 + \frac79\sqrt{15}\bigr)$, and the green line is $x=2y.$
[graph]zdzkxhk1dn[/graph]
[/sp]