If your pole is at 1 million volts and the ground is at zero, there is a potential difference between them.
Think of an electric line power line. The line is at 400,000V and the ground is your zero reference. The only reason it doesn't arc is because the path of least resistance is to follow the wire.
Now, if I get close enough to the wire, suddenly the path of least resistance is to arc through the air, through me and into the ground. And so it changes direction and follows that route instead - until I move away.