Here's another way. First, what is the area of the polygon with vertices at (x1, y1), (x2, y2), ...(xn, yn)? That's not as hard as it sounds. Just use green's theorem. You need to integrate y dx (or x dy) around the edges of the polygon, returning to your starting point (keeping the interior on your left side to have a positive orientation, or just taking the absolute value at the end). The integral from (xk, yk) to (xk+1, yk+1) is just:
[tex]\int_{x_k}^{x_{k+1}} \left( \frac{y_{k+1}-y_k}{x_{k+1}-x_k} (x-x_k) + y_k \right) dx= \frac{1}{2} (y_{k+1}+y_k)(x_{k+1}-x_k)[/tex]
So the total area is:
[tex]A=\frac{1}{2} \sum_{k=1}^{n} (y_{k+1}+y_k)(x_{k+1}-x_k)[/tex]
where we look at k (mod n), and after expanding and cancelling this simplifies to:
[tex]A=\frac{1}{2} \sum_{k=1}^{n} (x_k y_{k+1} - x_{k+1} y_k )[/tex]
Now if we look at four points on the unit circle, labelled by their angle in polar coordinates, [itex]\theta_k[/tex], the area of the corresponding inscribed quadrilateral is:<br />
<br />
[tex]A=\frac{1}{2} \sum_{k=1}^{4} \cos(\theta_k) \sin(\theta_{k+1}) - \cos(\theta_{k+1}) \sin(\theta_k) =\frac{1}{2} \sum_{k=1}^{4} \sin(\theta_{k+1}-\theta_k)[/tex]<br />
<br />
We clearly can't do any better than if the angles between adjacent vertices is pi/2 (90 degrees), so a square maximizes the area.[/itex]