The galaxies are moving, as you say with respect to each other.
When you say they are stationary, this means each is stationary with respect to its local environnent, and with respect to the universe as a whole : the universe looks as symmetric from either viewpoint.
Say we are in galaxy A looking at distant galaxy B. B is surrounded by other nearby galaxies B1 B2 etc... On average, from our perspective, B, B1, B2 share the same motion relative to us.
Which is the same as saying that from B's perspective, the group B1, B2,... has zero overall motion, or that B is stationary with respect to that group.
Actually, to better accuracy B will see itself at the center of a symmetrical universe and it is stationary with respect to that.
The "dots on an expanding balloon" image says it all better : look at the motion of the dots, forgetting the balloon. This is exactly how it works. Each dot moves away from other dots, and each is stationary with respect to the center of the group of surrounding dots.
http://www.phinds.com/balloonanalogy/