Here is the derivation mentioned in post #30.
Refer to the figure on the right. It shows half the hexagon as a quadrilateral inscribed in a circle of radius ##R##. The full hexagon can be obtained by reflecting the quadrilateral about the diameter or by appending to it itself rotated by ##180^{\circ}##.
I have labeled the angles subtended by the chords, by Greek letters related to their label in Latin, e.g. angle ##\alpha## for chord ##a## etc. I state without proof the following.
1. The angle subtended by a chord is invariant if the point that subtends it on the circumference is moved along the circumference. For example, ##\alpha = \angle DCA=\angle DBA.##
2. The angle subtended by a chord at the circumference is half the angle subtended by the same chord at the center of the circle. For example, the angle subtended by a diameter is ##180^{\circ}## at the center and ##90^{\circ}## at any point on the circumference.
3. The length of a chord is given by the the diameter times the sine of the subtended angle. For example,##~DA=a=2R\sin{\alpha}.##
Now for the derivation. We write the diagonal ##DB## in two ways using the law of cosines for triangle ##(DAB)## and the Pythagorean theorem for right triangle ##(DBC).## Note that the angle between vectors ##\mathbf a## and ##\mathbf x## is ##\theta = 180^{\circ}-(90^{\circ}+\beta)=(90^{\circ}-\beta)##. Therefore, ##\mathbf a\cdot\mathbf x=ax\sin\beta.## Thus, $$d^2=a^2+x^2+2ax\sin\beta=4R^2-b^2.$$ We now set ##x=R##, ##\sin\beta=\dfrac{b}{2R}~## and rearrange to get, $$ a^2+ab+b^2=3R^2.$$ Using the identity ##a^3-b^3=(a-b)(a^2+ab+b^3)## and rearranging, yields $$R^2=\frac{a^3-b^3}{3(a-b)}.$$While thinking about this, I stumbled upon something that might be of interest. I will post it on a separate thread on this forum as a solved problem.