@Keita what you have noticed is the distinction between what are sometimes called "active" vs "passive" transformations.
Passive: If you view the transformation as rotating the (x-y axes of) the coordinate system, but keeping the vectors themselves fixed, then the position vector and the matrix have new components in the new coordinate system and ##z' = \tfrac{1}{2}(\mathbf{x}')^T M' \mathbf{x}' = \tfrac{1}{2} \mathbf{x}^T R^T M' R \mathbf{x}##, leading to ##M' = R M R^T## after setting ##z' = z##.
Active: if you view the transformation as rotating the vector ##\mathbf{x} \mapsto R\mathbf{x}##, but keeping the coordinate system fixed, then obviously ##\mathbf{x}'## has different components to before but the matrix ##M## is unchanged. Then ##z = \tfrac{1}{2} (\mathbf{x}')^T M \mathbf{x}' = \tfrac{1}{2} \mathbf{x}^T R^T M R \mathbf{x}##. You could view this instead as a transformation of the operator itself as ##M \mapsto M' = R^T M R##.
The two different versions of ##M'## are related by transpose (because as you should be able to see: rotating the coordinates ##n## degrees clockwise is the same as rotating the vectors ##n## degrees anti-clockwise. )