In this context, you could say that potential is simply potential difference from a point to the reference point or ground, i.e., the point in your circuit where you define the potential to be zero. Potential difference is the difference in potential between two given points, neither one of which is the reference point
Explaining this with an analogy using gravity, things would go something like this.
Let's say we are standing at ground level and so we declare the ground to have zero potential energy. Ground is our reference, any mass that we raise above ground level will acquire a potential energy of mgh. mgh is your potential. If you have two of the same objects at difference heights, the potential difference would be mgh1 - mgh2, if h2 is lower than h1, then you have a positive potential difference...this could drive a "current" from point at h1 to point in h2...in other words, if you take a table and raise one end to a height of 24 inches and the other end to a height of just 12 inches...and you put a ball on the higher end...will it flow to the lower (potential) end? ...what if you put the ball in the lower end, will it climb to the higher end all by itself?
so, yes, current flows from the high potential to the low potential.
And not just current, also heat flows from high temperature to low temperature.
And, so, I simply keep a short statement in my head to remind me of this: Power flows downhill.