If you imagine the case of cavity being at the center of insulated sphere, the field is zero by symmetry. Use superposition to solve the problem. The fiels is indeed independent of the charge density of the sphere and the size of the cavity.
Surendranath