Why do magnets in a Gauss gun exhibit backward drift after being hit?

[Aleks]

So I was building a Gaussian gun for physics filming it at 300fps. See video:
http://youtu.be/hZcAk3uBp58

While skipping frame by frame, I realized that all the magnets started drifting backwards after being hit. Can someone explain in detail what is going on? Sorry if my physics is not advanced enough here, but it's either something really simple or I just broke physics.

I'm sure you're all familiar with the Gauss Gun, but if not - see one of my (attached) slides showing a dumbed down version of what it is.

 

[Bandersnatch]

That's just recoil. Aka conservation of momentum.