A slight touch wouldn't do anything. The pole is still extremely massive. And in equilibrium. To move it would require a force to overcome that massive inertia.
Hmm, I suppose that makes sense. But isn't the equilibrium a very very unstable one? I imagine it like balancing a straw on an orange. Once you have it there, even the slightest touch makes it roll off. Of course, a straw isn't really a good example of a very massive object, but still :p