I don't think the plates have to be conducting. They just have to respond to the electromagnetic field. In general, the Casimir, or Casimir-Polder forces are those that arise between objects through the exchange of virtual photons, so for example between an atom and a dielectric surface.
The calculation of the Casimir force is most easily done for conducting plates, because their presence only amounts to boundary condition on the field - it must vanish on the plates. The finite gap between the plates then 'selects' only discrete spectrum of field fluctuations. Since the spectrum changes, the total energy of the system changes as well (even though the total energy is not well-defined and one must use regularization) and most importantly the energy depends on the plates separation, so we have a force between the plates!
For general objects, the calculation of the Casimir-Polder force can look quite different.