The battery has one terminal at a potential that is higher than the other. What's happening here? Chemical reactions are due to the exchange of electrons between chemicals. When this results in one chemical gaining an excess of electrons and the other a deficit, then a potential difference will exist between the two chemicals. If you attach a copper wire connecting the terminal with an excess of electrons to the terminal with a deficit, then a current will flow to eliminate the discrepancy. If you don't connect a copper wire an electric field will exist, external to the battery, between the two terminals. Electrons can be moved from one place to another to create an excess in one place and a deficit in another by mechanical means. This is the cause of static electricity, where you rub amber with fur, or ball point pen with jumper. Whenever we have an excess of electrons in one place and a deficit in another, then there will be a force impelling the excess to redistribute more evenly.
Just as mass falls through differences of height (in a gravitational field) so charge falls through differences of voltage (in an electrical field). Just as a mass raised in a gravitational field has potential energy, which it releases when it falls; so charge has potential energy in an electrical field, which it releases when it moves to a lower voltage. Volts is a measure of this potential energy. The Volt is just a snappy name for the Joule/Coulomb [i.e. kg.m2/coulomb.s2], that is energy per unit charge.