So it looks like those magnets are magnetized axially (one flat face is N and the other flat face is S). To contain that B-field, you need a u-shaped piece of ferrous metal that you can slip over the top and bottom faces that fits fairly close to both, and has the two outer pieces connected along one or 3 of the sides. I couldn't find a picture of one, so hopefully that makes sense. If you only do that "keeper" for the bottom magnet and not the top stack, the top stack will be attracted to the ferrous keeper below it.
What spacing are you wanting between the bottom magnet and the top stack? Are you wanting to go from repelling to no force (other than the force of gravity on the top stack of magnets)? Are you wanting to basically turn "on and off" the repulsion? Would going from repulsion to attraction also work?
BTW, having the keeper that close to such a strong magnet will likely require a fairly thick keeper wall, to avoid saturation. Once the keeper material saturates, any B-field above the saturation level leaks past the keeper and will still interact with the other magnets.