It is hardly a dumb question. I haven't thought about this until just now, but I can attempt an explanation. Do not take this as the truth until it is confirmed by someone else (but it certainly provides a plausibility argument until such time).
There are (in my mind) 3 aspects to this problem :
1. Ozone formation : Ozone formation from oxygen typically requires UV excitation. There is more UV insolation in the upper reaches of the atmosphere. In fact, the ozone thus formed is itself the most dominant absorber of UV. So, it is hard to form very much ozone in the lower atmosphere. (Also, as you go much higher, say into the mesosphere, the density of oxygen becomes so small that it is again hard to form ozone; the optimal height happens to be in the stratosphere.)
But what stops the ozone formed in the stratosphere from sinking down ?
2. Transport : Due to the density of ozone being about 50% more than air, you would expect the ozone to sink towards the Earth's surface. This may well be happening, but this is a very slow process even under still conditions. In the presence of disturbances in the air, the downward drift might be even slower. (Typically, in a still medium, the rate of sinking depends on the ratio between the excess gravitational energy - proportional to the excess density - and the average thermal energy of the molecules, as well as the intermolecular attraction - which is quite high due to its polarity.)
But you may argue that no matter how slow the rate of sinking, the ozone should eventually find its way to the bottom and accumulate there. So, why is it important that the rate of sinking is slow ?
3. Stability : Ozone is not a particularly stable molecule. Even in a clean system it has a half-life of only about 10 hours. In the presence of moisture, CFCs, acidic gases, etc. this hal-life gets reduced to less than an hour. So, as the ozone sinks (assuming it does...even if very slowly), it gets destroyed, and is rarely "alive" long enough to make it down very far.
