Visualize the electric field a different way and I think your understanding will clear up.
Electric field is nothing more then the tendency of like electrostatic charges to repel one another and unlike charges to attract. If you have a positively charged object and a negatively charged object the excess protons in the positively charged object will repel each other. Electrons in the negatively charged object will also try to repel each other. Electrons in the negatively charged object will be attracted to protons in the positively charged object and vice verse. This attraction is measured in volts. This attraction (voltage) causes electrons to move from the negative object to the positive object, it is also possible for protons to move from the positive object to the negative object. That would not happen very much in a solid medium but in a liquid, gaseous, or plasma medium it certainly can. The movement of the charges is called current and measured in amperes.
The idea that electric potential can exist independent of charged particles comes from the idea that voltage can be induced my a magnetic field in an otherwise neutrally charged object.
This idea becomes unnecessary if magnetic fields are understood as nothing more then the result of electric fields that are in motion with respect to one another. Here is a link that explains this concept very well...
http://physics.weber.edu/schroeder/mrr/MRRtalk.html
So with this bit of understanding magnetism is out the window and everything is understood as the interaction of charged particles. To my mind this is the simplest possible explanation. Everything is nice and tidy and self consistent. That's how I like it. :-)