[TIKZ][scale=0.75]
\coordinate [label=above: $P_1$] (P1) at (90:5cm) ;
\coordinate [label=above right: $P_2$] (P2) at (33.13:5cm) ;
\coordinate [label=right: $P_3$] (P3) at (0:5cm) ;
\coordinate [label=below right: $P_4$] (P4) at (340:5cm) ;
\coordinate [label=below: $P_5$] (P5) at (270:5cm) ;
\coordinate [label=below left: $P_6$] (P6) at (213.13:5cm) ;
\coordinate [label=left: $P_7$] (P7) at (180:5cm) ;
\coordinate [label=above left: $P_8$] (P8) at (160:5cm) ;
\coordinate [label=left: $O$] (O) at (0,0) ;
\draw (P1) -- (P3) -- (P5) -- (P7) -- cycle ;
\draw (P2) -- (P4) -- (P6) -- (P8) -- cycle ;
\draw (P1) -- (P2) -- (P3) -- (P4) -- (P5) -- (P6) -- (P7) -- (P8) -- cycle ;
\draw (O) circle (5cm) ;
\fill (O) circle (2pt);
\draw[dashed] (P3) -- (O) -- (P2) ;
\draw[dashed] (P1) -- (O) -- (P4) ;
\draw[dashed] (O) -- (P5) ;
\draw (1,0.3) node {$\theta$} ;
\draw (1,-0.15) node {$\phi$} ;
[/TIKZ]
The square $P_1P_3P_5P_7$ has area $5$, so it has side $\sqrt5$ and diagonal $\sqrt{10}$. The diagonal of the square is the diameter of the circle, which therefore has radius $\sqrt{5/2}.$
If the sides of the rectangle $P_2P_4P_6P_8$ are $a$ and $b$, then $ab=4$ and (by Pythagoras) $a^2+b^2 = 10.$ Therefore the sides are $\sqrt2$ and $2\sqrt2.$ It follows that $\tan(\angle P_2P_6P_4) = \frac12.$
Let $\alpha = \angle P_2OP_4$. Then $\alpha = 2(\angle P_2P_6P_4).$ So $\tan\frac\alpha2 = \frac1 2$, from which $\sin\frac\alpha2 = \frac1 {\sqrt5}$ and $\cos\frac\alpha2 = \frac2{\sqrt5}.$
Let $\theta = \angle P_2OP_3$ and $\phi = \angle P_3OP_4$, as in the diagram, and notice that $\theta+\phi = \alpha.$
The area $| P_2OP_3|$ of the triangle $ P_2OP_3$ is $\frac54\sin\theta$ (remembering that the radius of the circle is $\sqrt{5/2}$). Similarly $| P_3OP_4| = \frac54\sin\phi.$ Also, $|P_1OP_2| = \frac54\sin\bigl(\frac\pi2 - \theta\bigr) = \frac54\cos\theta$, and $|P_4OP_5| = \frac54\cos\phi.$
The sum of the areas of those four triangles is the area of the right half of the octagon. The left half of the octagon is congruent to the right half and therefore has the same area. So the are of the whole octagon is $\frac52\bigl(\sin\theta + \sin\phi + \cos\theta+ \cos\phi\bigr).$
Now apply the "
sum-to-product" trig formulas to write that as $$5\Bigl(\sin\tfrac{\theta+\phi}2\cos\tfrac{\theta-\phi}2 + \cos\tfrac{\theta+\phi}2\cos\tfrac{\theta-\phi}2\Bigr) = 5\bigl(\sin\tfrac\alpha2 + \cos\tfrac\alpha2\bigr)\cos\tfrac{\theta-\phi}2 = 5\bigl(\tfrac1{\sqrt5} + \tfrac2{\sqrt5}\bigr)\cos\tfrac{\theta-\phi}2 = 3\sqrt5\cos\tfrac{\theta-\phi}2.$$ That is obviously maximised when $\theta=\phi$ so that $\cos\tfrac{\theta-\phi}2 = 1$. In conclusion, the maximum area of the octagon is $3\sqrt5$.