The electrons are bound to atoms with different energies. Many electrons will move, but then Coulomb repulsion (force) prevents more from moving. One can increase the magnetic field which increases the number of electrons that can be 'forced' to the same end of the conductor. Turn off the magnetic field and the electrons rapidly return to the atoms from which they had escaped.
It's a bit like pushing a box up frictionless (or perhaps very low friction) slope. One pushes the box up the gravity field so far and then stops. One still has to push against the force of gravity even when stopped. Then remove the pushing force and the box slides down the slope. A somewhat crude analogy.