In addition to potential difference, one can also define an absolute potential, which is the potential w.r.t. infinity. In other words, we can define an absolute voltage based on the energy it takes to move a test charge from infinity to the point of interest.
For the problem of an infinite conducting sheet, it actually takes ZERO energy to move a test charge from infinity to any point on the sheet (by the definition of a perfectly conducting sheet). So the sheet in this case is in some sense "grounded" at infinity.
As f95toli pointed out though, the use of infinity as a reference point is no more significant that using the potential of the Earth, or any other potential for that matter. That is why, rather than bothering to try to set voltages w.r.t. infinity, we choose a common reference point that is most convenient to us, which is typically Earth.
Claude.