First, I need to make a disclaimer. The following statements come from my memories of when I was a graduate student 30+ years ago, working on my Ph.D. in experimental particle physics. I have not worked actively in research since then, only in teaching. Others are welcome to confirm or correct these statements as they apply to experimental particle physicists today. I would not be surprised if there were occasional exceptions to these statements back in those days, or today; nevertheless, these are my general impressions. Feel free to mentally insert weasel-words like "generally" or "usually" where appropriate.
1. For individual particles, or bound systems of particles (e.g. an atom, an atomic nucleus, even a proton or neutron being viewed as a collection of quarks), we always said simply "mass." In this context, "rest mass" and "invariant mass" serve only to distinguish from "relativistic mass", and we never used the concept of "relativistic mass," so there was no need to make the distinction. During at least the last 30-40 years, the concept of "relativistic mass" appears only in writings about relativity for laymen, and in some introductory textbooks, as far as I know.
2. For unbound systems of particles, as I noted in the previous post, we used the term "invariant mass". I think we did not say simply "mass" because there is no single coherent "object" here.
For example, my advisor was studying the production of Λ0 particles in neutrino interactions in a bubble chamber. The signature for this is a V-shaped pair of tracks appearing near a primary interaction, representing the proton and π- produced in the decay of the invisible Λ0. One of my tasks was to calculate the invariant mass of the p and π-, based on measurements of the momenta and energies of the tracks, and compare it to the known mass of the Λ0 (1115.7 MeV/c2) in order to confirm that they did indeed come from a Λ0.