I think the reduced-mass concept should be generalized to a matrix rather than to a scalar. If we consider a N-particle system where the particle positions x(i) are measured relative to the C.M., we have, for a conservative system, that the energy equation (f.ex.) may be written as:
1/2*MV^(2)+1/2*(v^(T)*Q*v)+U=const.
Here, M is the system's total mass, V is the speed of C.M, U is the potential energy; whereas v is the N-vector of particle velocities, v^(T)=(v(1),..,v(N))
(T is for transpose)
(v(i)=dx(i)/dt)).
Q is the N*N diagonal mass matrix, Q(j,j)=m(j), where m(j) is the mass of the j-th particle.
(The resulting product of velocities, f.ex. v(j)^(2) is the dot product if v(j) is a vector.)
We have, by definition of particle positions relative to C.M, m(i)*x(i)=0, where summing over i=1,..N is implied.
Hence, we may eliminate a particle (the N'th, f.ex.), from our system, and
we represent the other particles by their distances x(i,N) (i=1,..N-1):
x(i)=x(N)+x(i,N), i=1,..N-1, v(i,N)=dx(i,N)/dt, vrel^(T)=(v(1,N),...v(N-1,N))
The energy equation may now be rewritten as:
1/2*MV^(2)+1/2*(vrel^(T)*R*vrel)+U=const.
Here, R is the (N-1)*(N-1) reduced mass matrix with respect to particle N:
R(j,j)=r(j,j)=m(j)(M-m(j))/M
R(i,j)=R(j,i)=-r(i,j), r(i,j)=m(i)*m(j)/M (i not equal to j)
We see that r(i,j) is less than both m(i) and m(j).
Again, products of velocities should be regarded as inner products if the velocities v(i,N) are vectors.