My very limited understanding is that 'standard' Casimir force is interaction between idealized perfectly conducting surfaces and vacuum fields. And that this is modified for real materials by detailed treatment involving consideration of van der Waals interactions between media, but not primarily due to such.
From
http://en.wikipedia.org/wiki/Casimir_effect
"..This force has been measured, and is a striking example of an effect purely due to second quantization.[3][4] However, the treatment of boundary conditions in these calculations has led to some controversy. In fact "Casimir's original goal was to compute the van der Waals force between polarizable molecules" of the metallic plates. Thus it can be interpreted without any reference to the zero-point energy (vacuum energy) or virtual particles of quantum fields.[5].." Further down, under 'More recent theory':
"A very complete analysis of the Casimir effect at short distances is based upon a detailed analysis of the van der Waals force by Lifgarbagez.[11][12] Using this approach, complications of the bounding surfaces, such as the modifications to the Casimir force due to finite conductivity, can be calculated numerically using the tabulated complex dielectric functions of the bounding materials. In addition to these factors, complications arise due to surface roughness of the boundary and to geometry effects such as degree of parallelism of bounding plates.
For boundaries at large separations, retardation effects give rise to a long-range interaction. For the case of two parallel plates composed of ideal metals in vacuum, the results reduce to Casimir’s.[13]"
But will stand corrected on this.