The simple way to answer this is to consider a universe containing only one electron. Here there will be an electric field and a potential, but no positive charges. You can reverse the argument using a lone proton.
Another aspect of this is that you can contemplate electric fields and potentials without charges. Some people could say this is dubious because charges and currents are the sources of electric and magnetic fields. But, at least locally we can have electric field and potential without charge. Two examples are the single charge case given above (far from the charge), and electromagnetic waves far from the source. Note that the first case can be static, but the second case must be dynamic which requires more careful consideration of the meaning of potential.