With the retarded van der Waals, the calculated forces are around an order of magnitude less than experiment. This was found out in the early 50's and motivated Lifgarbagez to derive his theory for dielectrics in '56. The microscopic method of using the corrected van der Waals is not a good means of calculating the Casimir force because van der Waals is not an additive force. In fact, I have not seen such a calculation or derivation, instead, we use macroscopic methods for calculation. Under these conditions, we must work with the macroscopic fields or equivalent sources of the fluctuating fields.
There are no charges of the like assumed to be explicitly involved in the vacuum explanation of the Casimir force. The vacuum state, while absent of photons, still has electric and magnetic fields. These fields fluctuate about a mean value of zero (hence why there are no observed fields in macroscopic ensemble of measurements). However, the energy density of these fields is infinite, only bounded by the frequency limit specified on electromagnetic waves. And, just like in many other examples in physics, if we disturb the energy density of the vacuum, the spatial dependency of this disturbance will cause a force.
The Casimir energy and force can be directly calculated using such assumptions. The introduction of scatterers into the vacuum forces the fluctuating fields to conform to the boundary conditions of the scatterers. In doing so, we restrict the number of modes that can exist in the vacuum state. Thus, the modes of the system of interest are now a subset of the original vacuum state. By quantifying the change in the energy between the original and desired systems we get the Casimir energy. To get the Casimir force, we can find the gradient of the Casimir energy (or by some equivalent method). How we quantify the energy content can be done by calculating the modes of the system, like in Casimir's original parallel plate or Lifgarbagez' method, finding the energy density of the fluctuating fields, this can be done by using the Maxwell Stress Tensor to find the force, or by finding the energy contained in the equivalent sources that, when impressed upon and inside the scatterers, perfectly reproduce the fluctuating fields. The latter method is based off of Schwinger's source theory model that he applied to the problem in a series of papers.
As for temperature and dielectric dependence, it is just a simple extension to these theories. Lifgarbagez' and Schwinger both present ways of dealing with arbitrary materials and temperatures.
Another way that gives equivalent results is to calculate the radiation pressure of the vacuum's virtual photons. This is to be expected of course since it would be equivalent to dealing with the energy density of the fluctuating fields directly.