The change-of-basis identity:
[tex]
\rho_{k' l'} = \sum_{m n}{\langle k' \vert m \rangle \, \rho_{m n} \, \langle n \vert l' \rangle}[/tex]
with the identification of the matrix:
[tex]
U_{n l'} \equiv \langle n \vert l'\rangle[/tex]
can be rewritten as:
[tex]
\mathbf{\rho}' = \mathbf{U}^{\dagger} \cdot \mathbf{\rho} \cdot \mathbf{U}[/tex]
where we used the fact that:
[tex]
\hat{U}^{\dagger}_{k' m} = U^{\ast}_{m k'} = (\langle m \vert k' \rangle)^{\ast} = \langle k' \vert m \rangle[/tex]
The unitarity of the similarity transformation matrix U is an expression of the orthonormality of the old and new bases:
[tex]
\langle m \vert n \rangle = \sum_{k'} {\langle m \vert k' \rangle \, \langle k' \vert n \rangle} = \sum_{k'} {U_{m k'} \, U^{\dagger}_{k' n}} = \left[\mathbf{U} \cdot \mathbf{U}^{\dagger}\right]_{m n} = \delta_{m n} \Rightarrow \mathbf{U} \cdot \mathbf{U}^{\dagger} = \mathbf{1}[/tex]
[tex]
\langle k' \vert l' \rangle = \sum_{m} \langle k' \vert m \rangle \, \langle m \vert l' \rangle = \sum_{m} U^{\dagger}_{k' m} \, U_{m l'} = \left[ \mathbf{U}^{\dagger} \cdot \mathbf{U} \right]_{k' l'} = \delta_{k' l'} \Rightarrow \mathbf{U}^{\dagger} \cdot \mathbf{U} = \mathbf{1}[/tex]