There's at least a couple ways to answer that question.
One mathematical way is that for every force dependent on position expressible as a gradient, there is an associated potential energy.
However, the simplest is to say that potential energy refers to all forms of energy that are not kinetic (yes, really), or due to [itex]E_{0}=mc^{2}[/itex].
Different forms of potential energy can be discovered by using conservation of total energy, and seeing what's missing (e.g., the potential energy stored in chemical bonds giving rise to explosive chemical reactions).