What you have there is the energy in the electrostatic field of the sphere. This one you need first. You get it from the corresponding Maxwell equations for static fields,
[tex]\vec{\nabla} \cdot \vec{E}=\frac{\rho}{\epsilon_0}, \quad \vec{\nabla} \times \vec{E}=0.[/tex]
[The above is corrected for typos according to #3; thanks for pointing them out!]
From the second equation you see that the electrostatic field is a potential field, i.e.,
[tex]\vec{E}=-\vec \nabla \Phi.[/tex]
Substituting this into the 2nd equation yields\
[tex]\Delta \Phi=-\frac{\rho}{\epsilon_0}.[/tex]
Now you have
[tex]\rho(\vec{x})=\begin{cases} \rho_0=\frac{3Q}{4 \pi R^3} & \text{for} \quad r=|\vec{x}<R|\\ 0 & \text{for} \quad r \geq R. \end{cases}.[/tex]
Due to symmetry, [itex]\Phi[/itex] depends only on [itex]r[/itex] and thus writing the Laplacian in spherical coordinates for [itex]r<R[/itex] leads to
[tex]
\frac{1}{r} \frac{\mathrm{d}}{\mathrm{d} r^2} [r \Phi(r)]=-\rho_0.[/tex]
From this you get
[tex]\epsilon_0 [r \Phi(r)]''=-\rho_0 r \; \Rightarrow \; \epsilon_0 [r \Phi(r)]'=-\frac{\rho_0}{2} r^2+C_1 \; \Rightarrow \; \Phi(r)=-\frac{\rho_0}{6 \epsilon_0} r^2+C_1+\frac{C_2}{r}.[/tex]
Since for [itex]r=0[/itex] the potential should be regular, we must have [itex]C_2=0[/itex].
For [itex]r>R[/itex] the same calculation leads to
[tex]\Phi(r)=\frac{C_3}{r},[/tex]
where we have determined one constant such that [itex]\Phi(r) \rightarrow 0[/itex] for [itex]r \rightarrow \infty[/itex]. The constant, [itex]C_3[/itex], is determined by Gauss's Law to be [itex]C_3=Q/(4 \pi \epsilon_0)[/itex].
Continuity at [itex]r=R[/itex] leads to
[tex]-\frac{Q}{8 \pi \epsilon_0 R}+C_2=\frac{Q}{4 \pi \epsilon_0 R}[/tex]
i.e.
[tex]C_2=\frac{3 Q}{8 \pi \epsilon_0 R}.[/tex]
For the total energy of the electric field you find
[tex]\mathcal{E}=\frac{\epsilon_0}{2} \int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \vec{E}^2(\vec{x})=\frac{1}{2} \int_{\mathbb{R^3}} \mathrm{d}^3 \vec{x} \rho(\vec{x}) \Phi(\vec{x}).[/tex]
This gives from our above result
[tex]\mathcal{E}=2 \pi \rho_0 \int_0^R \mathrm{d} r \Phi(r) \; r^2 =\frac{3Q^2}{20 \pi \epsilon_0 r}.[/tex]
In your solution, obviously a factor of [itex]4 \pi[/itex] is missing.