The notion of mass is indeed quite different in Newtonian compared to special-relativistic theory. It's more simple in the relativistic case ;-).
Anyway, the argument starts with the formulation of quantum theory in Minkowski or Galilei space-time, which leads you to investigate the unitary ray representations of the corresponding parts of the groups that are continuously connected to the identity, i.e., the proper orthochronous Poincare and (inhomogeneous) Galilei groups, respectively.
In the case of the Poincare group you'll figure out that mass is a Casimir operator of the corresponding Lie algebra given by the relation, m^2=p_{\mu} p^{\mu}, where p^{\mu} are the generators of the space-time translations. The further analysis turns out that any ray representation is induced by a unitary representation of the covering group of the Poincare group, i.e., instead of the proper orthochronous Lorentz group \mathrm{SO}(1,3)^{\uparrow} you use its covering \mathrm{SL}(2,\mathbb{C}). In this way you come to massive, massless and tachyonic representations. The latter seem not to lead to a sensible physical theory (except for non-interacting tachyons, but these are useless because not observable).
In the case of the Galilei group, it turns out that there are non-trivial central extensions of the group, and mass is the central charge. There is no physically sensible unitary representation of the Galilei group or it's covering group, but only the central extension with the non-zero mass as the central charge.
The subtle difference is that this implies a mass-superselection rule, i.e., there cannot be superpositions of states with different mass, which you don't have in relativistic quantum theory. The latter possibility is realized in nature on the elementary-particle level by the neutrinos, which always are produced in flavor eigenstates which are mixtures of mass eigenstates with different masses, leading to the well-established neutrino oscillations.