I am working through Ballentine's Quantum Mechanics books, and I am stuck... I would be appreciative if someone could walk me through this section with some help.

Here is my first confusion. In this case then [itex]|a_{m'}b_{n'}\rangle\langle a_{m'}b_{n'}|[/itex] do not form a complete set? Second, why is [itex]\rho[/itex] still in there? Whenever I have dealt with the state operator before, it is represented either by
[tex]
\rho=\sum_n\rho_n|\psi_n\rangle \langle\psi_n| \quad \text{or} \quad \rho=|\psi\rangle\langle\psi|
[/tex]
does that mean that the [itex]\rho[/itex] above is (are) the coefficients in the sum I just wrote above?

I have some more questions on the same page just after this part, but perhaps if I can get this, the rest will fall into place.

Sure they do. That expression in the middle with sum over m',n' is just a
resolution of unity. Cf. Ballentine's eq(1.26).

No, [itex]\rho[/itex] is a state operator. The general requirements for an operator to be a state operator are given in eqs 2.6, 2.7, 2.8.

Your [itex]|\psi_n\rangle[/itex] vectors correspond to the [itex]|a_i b_j\rangle[/itex] in the tensor product case.

Try writing your [itex]\rho[/itex] as a double sum, generalizing the single sum you wrote above. Then substitute it into the expression for [itex] Tr(\rho R) [/itex] and see what you get.

Okay, starting with
[tex]\rho=\sum_m\sum_n\rho_{mn}|a_mb_n\rangle \langle a_mb_n|
[/tex]
then
[tex]
\langle R \rangle = \text{Tr}(\rho R)=\text{Tr}\left(\sum_m\sum_n\rho_{mn}|a_mb_n \rangle \langle a_mb_n|R\right)=\sum_m\sum_n\rho_{mn}\langle a_mb_n|R|a_mb_n\rangle
[/tex]
Then using the fact that [itex]|a_mb_n\rangle\langle a_mb_n|[/itex] form a complete set
[tex]
\langle R \rangle=\sum_m\sum_n\sum_{m'}\sum_{n'}\langle a_mb_n|\rho_{mn}|a_{m'}b_{n'}\rangle\langle a_{m'}b_{n'}|R|a_mb_n\rangle
[/tex]
Are these the steps Ballentine is taking?

I think that's only correct in general if the [itex]|a_mb_n\rangle[/itex] are eigenstates of [itex]\rho[/itex]. In a more general basis, the state operator is not diagonal. So my hint about "double-sum" was too easy to misinterpret, sorry.

I don't think you're taking the trace correctly here. But it's probably quickest if I just
show you the following:

For the simpler case of a non-composite system, in an arbitrary basis
[tex]
\def\<{\langle}
\def\>{\rangle}
\<R\> = Tr(\rho R) = \sum_j \<j| \rho R |j\> = \sum_j \<j| \rho 1 R |j\>
= \sum_{jk} \<j| \rho \; |k\>\<k| \; R |j\>
[/tex]

Can you generalize that to Ballentine's 2-component composite system where the basis vectors are Ballentine's eq(8.11) ? It should give you eq(8.12) almost trivially.

what you wrote makes it very easy to see the result thanks. My confusion is the above step. I had only ever seen [itex]\rho=|\psi\rangle\langle\psi|[/itex] or the way earlier with the sum. And I thought that when taking the trace, [itex]\rho[/itex] was expressed in terms of it's basis vectors such as in (2.14) and (2.15). I understand that section is on pure states, but rho is expressed in terms of it's basis vectors, not kept as rho. I thought I had to do that every time.

Thanks for your help. I'll try and finish out the section and if I have more questions I'll put them up.