It's actually fairly simple. I haven't worked it all the way out to the bitter end, but I've verified my method and it works.
Expand [tex]\sin 5\theta[/tex] = [tex]\sin(3\theta + 2\theta)[/tex] in terms of [tex]\sin \theta[/tex]. You would just use the addition formula followed by expanding each of the terms with the triple and double angle formulae.
All the terms with powers of [tex]\cos\theta[/tex] will have even powers of the cosine, so they can easily be converted to even powers of sine with [tex]\cos^2\theta = 1 - \sin^2\theta[/tex].
You will eventually get a reducible quintic in terms of sine theta. Let [itex]s = \sin\theta[/itex]
[tex]\sin 5\theta = 16s^5 - 20s^3 + 5s[/tex]
Equate that to zero (since [itex]\sin\pi = 0[/itex]) and solve.
Dismissing [itex]s = 0[/itex], it becomes a quartic which is actually a quadratic in [itex]s^2[/itex], which you can solve to get :
[tex]s^2 = \frac{1}{8}(5 \pm \sqrt{5})[/tex]
One of the values (with the plus sign is a redundant root).
EDIT : I've not yet found a way to determine the explicit value of [itex]s[/itex] from that expression. I keep getting ugly expressions with more roots of surds. I'm still working on this part, but for now, my answer is :
[tex]\sin{\frac{\pi}{5}} = \sqrt{\frac{1}{8}(5 - \sqrt{5})}[/tex]