A Lie group is called compact if it is compact as a manifold. For example, [itex]SU(2)[/itex] is compact because (topologically) it can be identified with the 3-sphere [itex]S^{3}[/itex] which is compact. The Lorentz group [itex]SO(1,3)[/itex] is not compact because it can “essentially” be written as a product of the non-compact space (of boosts) [itex]\mathbb{R}^{3}[/itex] with the compact space (of rotations) [itex]S^{3}[/itex]: [tex]SO(1,3) \cong SL(2 , \mathbb{C}) / \mathbb{Z}_{2} \cong \mathbb{R}^{3} \times S^{3} / \mathbb{Z}_{2} \ .[/tex]
The proper mathematical proof goes as follow: Recall that a subset [itex]\mathcal{U}[/itex] of [itex]\mathbb{R}^{n}[/itex] is compact if and only if, it is closed and bounded, i.e., if and only if, for every sequence [itex]a_{m} \in \mathcal{U} \subset \mathbb{R}^{n}[/itex], there exists a subsequence which converges to some [itex]a \in \mathcal{U}[/itex]. For simplicity, consider the 2-dimensional Lorentz group [itex]SO(1,1)[/itex]. Define a sequence of elements [itex]\Lambda_{m} \in SO(1,1)[/itex] by [tex]\Lambda_{m} = \begin{pmatrix} \cosh (m) & \sinh (m) \\ \sinh (m) & \cosh (m) \end{pmatrix} \ .[/tex] Now, since the components of [itex]\Lambda_{m}[/itex] are unbounded, it follows that [itex]\Lambda_{m}[/itex] cannot have convergent subsequence. Thus, [itex]SO(1,1)[/itex] is a non-compact Lie group.