It is not the amount of work needed to move a coulomb charge but how likely it will move a coulomb charge and with how much energy when it reaches the other end. I might be wrong but i believe that Electrical Potential Difference borrows the concept of Gravitational Potential Difference.
Imagine the this, you have 2 rocks with the same mass. One is displaced 5m above ground and another one 10m above ground.
-Which one is more likely to fall down?
Both are going to fall down since the potential difference is not zero.
-Which one will have more energy when it reaches the ground?
The one with higher potential or being at 10m will have more energy when it hits the ground.
-What if there is a table in between the floor and the ground with 5m of height?
If there is a table at 5m, it will become the new ground for the potential difference. Meaning that the rock at 10m will only fall 5m and have less potential energy when it reaches the table. And the rock at 5m will not fall because it is already at the table making it with 0 potential difference.
Now do you get why it is called potential difference? The same concept can be applied to electricity except that the attraction are not done by mass but the charges. And the other thing is that Electrical Potential Difference not only provide the force for pull but also for push.