That is the part that confuses me. While I accept that we only keep the continuous transformations as part of relativity, I never got a good physical feeling for why.
I'll try to write something now, although I can not spend much time into details yet.
The tools we use to study Lie algebra representations do not know about the "large" topological structures of the corresponding Lie group. In Lie algebra theory, to obtain a finite transformation one exponentiate the generators times the parameters of the transformation. Another way to say, when the (parameters of the) transformation (are) is infinitesimal, the generators appear just as linear terms exp[i x G ] = 1 + i x G + ... where x is the parameter (angle) and G the generator. We study representations of Lie groups using the Lie algebra, the commutation relations between the generators [Gi,Gj]. For a complicated non-trivial topology, one will obtain representations which can be decomposed into simple representations.
The short answer is essentially : the tools from Lie algebra give us only the connected part to the identity (because det [ exp^{ i * x * G } ] =1) so when we have parts which are not continuously connected with identity (such as when we have time reversal or parity), we obtain additional discrete topological quantum numbers which are dealt with individually.