Wow, I forgot this was even here.
I'll post a solution, since I don't like unanswered threads.
The electrostatic self-potential energy is the amount of work required to increase the radius of a spherically symmetric charge distribution from r to r+dr. From the definition of voltage, $$dW=Vdq$$. The charge dq is $$4 \pi \rho r^2$$. The potential V is $$\frac{kq}{r}=\frac{4 \pi \rho k r^3}{3r}=\frac{4 \pi \rho k r^2}{3}$$. $$k$$ is the Coulomb's Law constant, $$k=\frac{1}{4 \pi \epsilon_0}$$. Multiplying these two expressions, we have $$\frac{16 \pi^2 \rho^2 k r^4}{3}$$. Integrating from 0 to R (increasing the radius from 0 to R by putting on more and more shells), we have
$$\frac{16}{3} \pi^2 \rho^2 k \int_0^R r^4 dr=\frac{16}{15} \pi^2 \rho^2 k R^5[/math] as our answer.