Pick up a book on Lie groups. Especially something like a book on group theory for physics/physicists. But here is a summary on the classical groups:
The classical groups are denoted by the dimension and algebraic type of the spaces on which they act in their principle representation:
<Group Type>(dimension;number field) or when the space has some indefinite character: <Group Type>(+dimen, -dimen; number field).
There are four types Linear, Orthogonal, Unitary, and Symplectic.
LINEAR GROUPS: Begin with a set of NxN square matrices. Take the invertible ones and you get the group GL(N) the (G)eneral (L)inear group. Depending on whether you allow real or complex matrices you get GL(N;R) or GL(N;C) respectively. Now This group has a subgroup of simple scaling transformations i.e. the elements which are non-zero multiples of the identity matrix. Taking these types out you get the (S)pecial (L)inear groups, SL(N;R) or SL(N;C).
We can then define the other classical groups as subgroups of the real or complex linear groups by imposing conditions. If you start with the special linear group instead of the general case an S (for special) is put in the beginning of the group name. Thus for Orthogonal you have O(N) and SO(N) and for unitary groups you have U(N) and SU(N). Symplectic groups are already "special" so no prefix is required (but they still start with S so it all works out nicely.)
ORTHOGNAL GROUPS: If you restrict your matrices to those which preserve a specific bilinear form i.e. a metric then you get an orthogonal group. The metric must be non-singular but it may be indefinite i.e. some vectors on which the matrices act may have positive square norm and some may have negative square norm. In general you can diagonalize the metric to have say p positive dimensions and n negative dimensions. The corresponding orthogonal subgroups of GL(N;R) and SL(N;R) are O(p,n) and SO(p,n) respectively (note that p+n=N). If you allow complex vectors then the positive vs negative is meaningless and you simply write SO(N;C). Sometimes in the indefinite cases we qualify the groups/group-elements as being pseudo-orthogonal instead of simply orthogonal. But this isn't universal. The special cases SO(N,0;R) and SO(0,N;R) are usually written simply as SO(N).
Orthogonal transformations are rotations or in the indefinite case we use the more general term pseudo-rotations.
SYMPLECTIC GROUPS: If instead of a symmetric bilinear form you have an anti-symmetric bilinear form (a symplectic form instead of a metric) then firstly the dimension must be even and secondly there are no "non-special" transformations. The group you get is Sp(N;R or C). There is also an intimate connection between the symplectic groups and the quaternions. I'll leave that to further reading. An example of symplectic forms comes up when you look at phase space (positions + momenta) in the canonical treatment of mechanics.
(pseudo-)UNITARY GROUPS: Finally with complex matrices there are the unitary subgroups which preserve a Hermitan form. This too can be indefinite (and in such case we may call them pseudo-unitary) so you have as subgroups of GL(N;C) the unitary groups U(p,n) and when restricting to special unitary transformation you get SU(p,n). The special cases of SU(N,0) and SU(0,N) are written simply as SU(N). The unitary groups most often arise in quantum theory where the hermitian form defines transition probabilities in experiments. One other point about unitary groups. Although they are traditionally represented by complex matrices the group in general is a real group in that real and imaginary components of the matrices are not treated as the same. You can futher "complexify" the unitary groups in which they become the same as the complex linear groups.
The preservation of various forms can also be expressed (in a basis dependent way) as conditions on the actual matrices. Specifically the orthogonal group is made up of orthogonal matrices which are defined by their transposes being also their inverses (for the definite cases when using an orthonormal basis). The unitary groups are made up of unitary matrices which have the property that their conjugate transposes are their inverses (again for the definite cases with orthonormal basis).
Symplectic matrices and pseudo-orthogonal or pseudo-unitary matrices (where the metric or H-form is indefinite) are a little bit tougher to express (in English) and so I'll leave it to you to read up on the subject. You can also consider quaternionic as well as real and complex matrices and their corresponding groups. For the quaternionic symplectic groups you can also have indefiniteness in the form hence you may see references to Sp(p,n) which like the unitary groups are actually real in the sense I described earlier.
One other point. The groups themselves are abstractions independent of the actual matrix representation you use. Thus some different types of groups are actually the same (isomorphic) or almost the same (have isomorphic Lie algebras, also called locally isomorphic). Some cases of locally isomorphic groups are:
SO(2;R) and U(1); SO(2;C) and GL(1;C);
SO(3) and SU(2) and Sp(2);
SO(3,1) and SL(2;C);
SO(6) and SU(4); SO(4,2) and SU(2,2)
and a few more but only a finite set.
These local isomorphisms between orthogonal groups and unitary, linear or symplectic groups relates to spinors.
The local isomorphism can be made a true isomorphism by adding one more restriction especially to the even orthogonal groups. You will then sometimes see the notation PSO(N) which stands for projective special orthogonal group. Thus for example PSO(6) = SU(4) where SO(6) is only "locally isomorphic to" SU(4). What happens here is that the negative of a vector is identified with the positive so a 180 degree rotation is equated with the identity transformation (You rotate pairs of dimensions 180 deg for all pairs given total dimension even). Also you may see the "projective" qualifier on unitary groups in the same way. There are slight variances in convention and definition in the literature.
Beyond finding a good basic text, if you want to get a deeper understanding of spinors look for a book on clifford algebras, e.g.: Porteus, "Clifford algebras and the classical groups". Also you might web search of find text referencing Cartan's classification of the simple Lie groups/ Lie algebras.
Cartan used a different notation with group/Lie algebra classes labeled A,B,C,D,E,F, and G and indexed by rank instead of dimension of a specific representation. These include some exceptional groups/Lie algebras beyond the classical ones.I've been simplistic on certain points but as you see it still it gets a bit involved. I hope this gives you a starting point. Understanding the classical groups, their Lie algebras, and the representations thereof is essential to understanding modern physics.
Regards,
James Baugh
[Edit] PS: I made a few errors with respect to projective orthogonal = unitary. The unitary cases have larger centers and so
one also define projective unitary groups: PSO(6) = PSU(4). And also since it is odd orthogonal SO(3) = PSU(2). There is an extra sign in the spin group corresponding to a 360deg rotation of the orthogonal and so it is the unitary groups on which we have the greater need to append a P to get the isometry.
R. J.B.
