A Lie group is called compact if it is compact as a manifold. For example, SU(2) is compact because (topologically) it can be identified with the 3-sphere S^{3} which is compact. The Lorentz group SO(1,3) is not compact because it can “essentially” be written as a product of the non-compact space (of boosts) \mathbb{R}^{3} with the compact space (of rotations) S^{3}: SO(1,3) \cong SL(2 , \mathbb{C}) / \mathbb{Z}_{2} \cong \mathbb{R}^{3} \times S^{3} / \mathbb{Z}_{2} \ .
The proper mathematical proof goes as follow: Recall that a subset \mathcal{U} of \mathbb{R}^{n} is compact if and only if, it is closed and bounded, i.e., if and only if, for every sequence a_{m} \in \mathcal{U} \subset \mathbb{R}^{n}, there exists a subsequence which converges to some a \in \mathcal{U}. For simplicity, consider the 2-dimensional Lorentz group SO(1,1). Define a sequence of elements \Lambda_{m} \in SO(1,1) by \Lambda_{m} = \begin{pmatrix} \cosh (m) & \sinh (m) \\ \sinh (m) & \cosh (m) \end{pmatrix} \ . Now, since the components of \Lambda_{m} are unbounded, it follows that \Lambda_{m} cannot have convergent subsequence. Thus, SO(1,1) is a non-compact Lie group.