Mathematically, wouldn't the loss of electrons increase voltage?

if you spend a work of 50 joules , this means you either spent the whole 50 joules on one electron and created a P.D. of 5o volts , or you spend them on two electrons and create a p.d. of 25 volts .

but by the way its not about electrons flowing from where the number is high to where the number is low , its about overcoming the potential difference even if the number of electrons is the same( more electrons usually mean more potential difference but thats not always the case ) .
in case of batteries , its not about equalizing the number of electrons , its about equalizing the potential difference across the poles of a battery , also i find it better not to think of the electric current in terms of electrons , think of it in terms of charge . electrons are weird little things !

 

Think of it as a two-step process.

1. The charge distribution creates an electric field, which is equivalent to voltage difference between the sides.

2. In the presence of this electric field, the charges experience a force which makes them move from higher voltages to lower voltages.

There is an interdependency between the charge positions and the electric field. One depends on the other, which depends on the other....


Now voltage is DEFINED so that its downhill gradient points in the direction of the electric field (so 1 to 0 means the electric field points left to right). As the electrons move, the electric field that they create changes, and the voltage changes. Eventually, with 4 on each side, the voltage will be constant throughout the conductor. This is the steady state.

Voltage can be thought of in terms of the amount of Joules one Coulomb loses when going from e.g. 1 V to 0 V, but this isn't particularly useful when the voltage depends on where the Coulomb of charge is! Your case here, where the electric field generators (the charges) are moving, is an electrodynamic problem, so you need to be more careful when using this formula. Most of the time the electrostatic approximation is fine, charges either move so fast you assume they're always in a stationary state (e.g. in perfect conductors), or they don't move at all (e.g. in perfect insulators).

Your case is an electrodynamic problem because you're asking for the details of the electrons' movements as they equilibrate within a conductor. That means you can't assume the voltage distribution is constant.

Logically, try to think of it this way:

Charge distribution causes electric field/voltage
Electric field/voltage causes forces to act on charges
As these charges move in this field, they lose or gain energy.

The energy they lose/gain is the voltage times their charge.

So calculate the voltage, then the energy, not the other way around.