Because of the symmetry between the pairs $(a,d)$ and $(b,c)$, we may reduce the problem to one of two variables:
Prove:
$$a^2+b^2+ab^2<2$$
where $a$ and $b$ are found from the following diagram:
View attachment 971
Using the Law of Cosines, we find:
$$a^2=2-2\cos(\alpha)$$
$$b^2=2-2\cos(\beta)$$
Since $\alpha$ and $\beta$ are complementary, we now have the objective function in one variable (after dividing through by 2):
$$f(\alpha)=2-\left(\cos(\alpha)+\sin(\alpha) \right)+\sqrt{2-2\cos(\alpha)}\left(1-\sin(\alpha) \right)<1$$
Differentiating with respect to $\alpha$ and equating to zero, we may find the critical value(s) (using a numeric root-finding technique) on the relevant interval:
$$\alpha\approx1.06048703739259044748$$
Thus, checking the end-points of the interval, we find:
$$f(0)=1$$
$$f(1.06048703739259044748)\approx0.767829392509715815674068$$
$$f\left(\frac{\pi}{2} \right)=1$$
Thus, we may conclude that for $$0<\alpha<\frac{\pi}{2}$$ we must have:
$$f(\alpha)<1$$
which is what we needed to show.
