Actually, if you try this you will notice that the voltage drops back to zero quite quickly (seconds).
This is because the crystal is not a perfect insulator, and because you must extract some current to measure the voltage.
Yeah, when you deform a piezoelectric crystal, a voltage builds up. But that means that there is going to be a force acting on any free charges hanging around, which will try to rearrange themselves to cancel the voltage. So after a while the crystal should be neutral again.