If the point is rational: then the upper derivative is 0 and the lower derivative is negative infinity. (as you postulated)
However, if the point is irrational, then the upper derivative is infinity, and the lower derivative is 0. (for pretty much the same reason as in the rational case...