I don't have a complete answer to all your questions, but I think I can say a few things that will be helpful. Perhaps the most helpful thing I can say is this: You should check out chapter 2 of volume 1 of "The quantum theory of fields", by Steven Weinberg.
Gigi said:
I read that the generator of the O(3) group is the angular momentum L and that the generator of the SU(2) group is spin S.
...
1. In some books they say that the generator of the SO(3) group is angular momentum L.
SO(3) is the group of proper rotations, i.e. det(Matrix)=1.
Thus is it O(3) or SO(3)?
I don't see a reason to call the generators of SO(3) angular momentum operators and the generators of SU(2) spin operators. Don't those two Lie groups have the same Lie algebras? (I'm pretty sure they do). I would just call them spin operators.
The difference between O(3) and SO(3) is only that O(3) includes reflections. Every member of O(3) that isn't a member of SO(3) is just -1 times a member of SO(3). I would only use the word "rotation" about a member of SO(3).
The most important consequence of the difference between O(3) and SO(3) is that SO(3) is a connected Lie group and O(3) is not. If you think of a member of SO(3) as a function of three parameters (e.g. Euler angles), you can always reach the identity element by continuously changing the parameters. You can't do that with an arbitrary member of O(3).
Hm...is the word "generator" even used when we're talking about a Lie group that isn't connected? I doubt it.
Gigi said:
2. Both O(3) and SO(3) are defined as rotations in Eucledian space, 3-dimensions. Thus I would expect that we are talking about classic angular momentum.
Nevertheless in a quantum mechanics book I read that S0(3) is the generator of the angular momentum operator.
How is that if in Quantum Mechanics we are using the Hilbert space that is a complex function space?
States in QM are represented by vectors in a Hilbert space H. If one observer describes the system as being in state X, then a rotated observer will describe the system as being in state U(R)X, where R is a SO(3) rotation and U is a function from SO(3) into GL(H) (the set of linear operators on H). U is said to be a representation of SO(3) (if it satisfies certain conditions) and H is said to be the representation space. What Weinberg does in his book is basically to Taylor expand U(R) as a function of the Euler angles (or whatever parameters you choose), and what pops out as a result is the spin operators and their commutation relations.
The condition that U(R) must be unitary implies that the first-order terms in the Taylor expansion must be anti-Hermitian. There are three of them, since there are three parameters, so they can be written as i\theta_i J_i where the \theta_i are real numbers and the J_i are Hermitian. These J_i are the spin operators.
Gigi said:
3. I have the similar question regarding SU(2). SU(2) is defined in complex space. Thus it is ok to say that it is more or less the same as saying that this complex space is the Hilbert space?
No, it's not. The relationship between SU(2) and the Hilbert space H is that there must exist a function from SU(2) into GL(H) that satisfies certain conditions.
Gigi said:
4. Now in relativistic quantum mechanics, the underlying group is the Lorentz group. Would that mean that the O(3) and SU(2) groups reflect only symmetries in the non-relativistic world? i.e. Schroedinger's equation?
No, it doesn't. The Schrödinger equation is present in both relativistic and non-relativistic QM. You can find it by using the method I outlined above on the group of translations in time instead of on the group of rotations in space. The generator of translations in time is the Hamiltonian. It's fairly easy to show that the operator that tells us how an observer that's translated in time relative to us would describe the same state, must take the form exp(-iHt). This operator satisfies the Schrödinger equation, so the result of this operator acting on a state vector is a time-dependent state vector that satisfies the Schrödinger equation.
The group of translations in space
and time is of special importance. When we consider representations of that group we find not only the Hamiltonian, but also the momentum operators and their commutation relations, which imply that the invariant square of the four-momentum operator is a constant, which we can use to define the mass of the states the operators are acting on:
-m^2=P^\mu P_\mu=-H^2+\vec{P}^2
Expressed in terms of the eigenvalues of the operators, and in units where c isn't =1, this is just the familiar
E^2=\vec{p}^2c^2+m^2c^4
Gigi said:
If that is so, how come spin that is a relativistic effect is explained using the SU(2) group?
Spin is not a relativistic effect. The relationship between SO(3) and SU(2) is that if you take SU(2) and identify members with opposite sign, then you get a group that's isomorphic to SO(3). This is often expressed as SO(3)=SU(2)/Z
2. It has nothing to do with relativity. The reason QM deals with representations of SU(2) instead of representations of SO(3) is just that choosing SU(2) guarantees that the identity U(RR')=U(R)U(R') will hold instead of U(RR')=exp(it)U(R)U(R') where t is sometimes 0 and sometimes pi. If I understand Weinberg right (I hope someone will tell me if I don't), it's a matter of mathematical convenience, and it doesn't mess up any of the physics except that superpositions of integer spin and half-integer spin won't be explicitly forbidden.
If you instead of SO(3) consider the Lorentz group SO(3,1), the same method that told us that SO(3)=SU(2)/Z
2 now tells us that SO(3,1)=SL(2,C)/Z
2. SL(2,C) is the group of all complex 2x2 matrices with determinant 1. In relativistic QM one chooses to represent SL(2,C) instead of SO(3,1) for the same reason as the corresponding choice in non-relativistic QM.
The only reason why this explanation of spin is usually done in a relativistic framework is that it's not any more difficult to start with SO(3,1) instead of SO(3).