The point is, its wrong to say that "electric potential is the energy that a charge would posses". Potential energy(either electric or gravitational), is a property of all the participants in the interactions, not just a single one or even a group of them.
Another important point, is that the potential itself has no physical meaning and only the potential difference is significant which, through the definition of electric field as the spatial derivative of potential, means its the electric field that is significant. So its irrelevant whether potential is high or low, it just matters that the potential difference is enough or not.