The gravitational force inside a homogeneous sphere depends only on the mass closer to the center- the force vectors due to mass farther out cancel each other.
If the radius of the sphere is R, then its volume is [itex](4/3)\pi R^3[/itex] and the density is [itex](3/4)m/(\pi R^3)[/itex]. That means that the mass of the sphere below radius r is [itex](3/4)m/(\pi R^3)(4/3)\pi r^3= m(r/R)^3[/itex]. The force on an object of mass M then is [itex]-Gm(r/R)^3M/r^2= -Gmr/R^3[/itex].
So the force is proportional to r and the potential energy is [itex]-Gmr^2/(2R^3)[/itex].