cube137 said:
I read that states are positive operators of unit trace - not elements of a vector space.
Is it referring to quantum states or all classical states?
[...]quantum state is rays in Hilbert space or vectors of unit 1?
It refers to quantum states.
There are two different concepts of state, one a special case of the other, but phrased in a different language.
The general situation is that of a state described by a density operator ##\rho##, a positive semidefinite Hermitian operator of trace 1 on a Hilbert space . (They are elements of another Hilbert space, the space of all Hermitian operators with the trace inner product.)
If such a density operator has rank 1 then it is called a pure state and has the form ##\rho=\psi\psi^*## with a norm 1 vector ##\psi## in the first Hilbert space. This vector (called a state vector or a wave function) is only determined by the state up to a phase, whereas the associated ray, the 1-dimensional vector space spanned by ##\psi##, is determined uniquely (it is the range of ##\rho##). Thus pure states are equivalently described by a ray in a Hilbert space.
The description by rays is less general since most states (e.g., for an infinite-dimensional Hilbert space all states at finite temperature, and hence all preparable position or momentum dependent states) are not pure. Unpolarized states of a photon are also not pure.
However, pure states are often used as simplified idealizations of general states. For example, in much (not all) of chemistry, only the electronic ground state of molecules is significantly populated, and in the absence of degeneracy, this is a pure state. Quantum information theory also works mostly with pure states, and leaves the non-idealized situation to specialists in quantum optics.
cube137 said:
I know operators are like minus, plus, square root and vectors are like rays in Hilbert space.
This is not true. Operations like plus, minus, sqrt, are neither operators nor rays, and not even alike rays in any sense.
Vectors are also not rays but a ray is the collection of all vectors parallel to a given one.
One can add state vectors and gets another state vector, but adding two distinct rays produces a 2-diemnsional subspace and not a ray.