First, Casimir didn't "say" anything. He calculated. I think the distinction is important - theoretical physics is accomplished not by stringing words together in the right order, but through quantitative predictions based on solid calculations.
Second, it matters that the plates are conductors. The key to the calculation is that the field inside a conductor is zero.
Third, I think this uncovers a problem with learning physics from popularizations, which are written for people who have (maybe) had high school physics, but not much more. I was taught the Casimir effect half way through my second year of grad school: so we're talking 5-1/2 years of full-time study beyond where the book is pitched. This is a huge amount og material that's missing, and people are simply going to have to accept that any understanding that's missing this much background is going to be superficial.
