OK I solved this, so I might as well share it with everyone else.
First of all, for convenience, I changed the constraint to R RT=I, which is completely equivalent.
So the problem is as follows:
Maximize tr{R_{xy}R^{T}} subject to R R^{T}=I
The solution:
First of all, we can...