In relativity it's indeed the center-momentum frame, not the center of mass frame. That's because today 112 years after Minkowski's crucial article about the mathematical structure of special relativity ("Minkowski space") we express everything in covariant quantities rather than in some arbitrary confusing ones, and that's why "relativistic mass" is not used anymore anywhere in current research (though there are still some new textbooks introducing the confusion, because the authors are unwilling to learn century-old math ;-)).
The invariant mass of a system (e.g., a set of point particles or a continuum mechanics description of a fluid or some fields like the electromagnetic field) is given by the total four-momentum ##P^{\mu}## of the system by
$$M^2 c^2=P_{\mu} P^{\mu} = (E/c)^2-\vec{P}^2 \geq 0.$$
For ##M>0## you can always find an inertial frame, where ##\vec{P}=0##, and that's called the center-of-momentum frame, and it's considered as the "rest frame" of the system.
The reason is that from Noether's theorem applied to Lorentz boosts it follows that for a closed system the center energy-weighted average rather than the mass-weighted average moves with constant velocity.
E.g., take two interacting particles. Their total momentum is conserved (Noether's theorem applied to translation invariance in space and time), i.e.,
$$p_1+p_2=\text{const}.$$
Written in terms of the coordinate time that reads
$$m_1 \gamma_1 \dot{x}_1 + m_2 \gamma_2 \dot{x}_2=\text{const},$$
but
$$m_1 \gamma_1=E_1/c^2, \quad m_2 \gamma_2=E_2/c^2.$$
This implies
$$E_1 \dot{\vec{x}}_1 + E_2 \dot{\vec{x}}_2=\text{const}.$$
The temporal component means
$$E=E_1+E_2=\text{const},$$
and thus the energy-weighted average of the three-velocities (rather than the mass-averaged three-velocities) is conserved,
$$\vec{V}=\frac{E_1 \dot{\vec{x}}_1 + E_2 \dot{\vec{x}}_2}{E_1+E_2}=\text{const}.$$
