In modern physics (i.e., since 1908, when Minkowski discovered the mathematical structure behind the special theory of relativity) we are used to call mass the quantitity that's more accurately called "invariant mass". This is a quantity that is independent of the object's velocity and thus the same in any inertial frame of reference.
For a classical point particle, this invariant mass (which is the only sensible definition of mass one can think of) is defined as
$$p^{\mu}=m \frac{\mathrm{d} x^{\mu}}{\mathrm{d} \tau},$$
where ##x^{\mu}## is the particle's position in four-dimensional spacetime, i.e., ##x^0=c t## (##c##: speed of light, ##t## coordinate time wrt. to the frame of reference we work in, and ##\tau## the proper time of the particle). Since by definition of proper time
$$\frac{\mathrm{d} x^{\mu}}{\mathrm{d} \tau}\frac{\mathrm{d} x_{\mu}}{\mathrm{d} \tau}=c^2$$
we have
$$p_{\mu} p^{\mu}=m^2 c^2,$$
which shows that ##m^2## is a scalar quantity. Since ##p^0=E/c## this expression, split in temporal and spatial components, reads
$$\frac{E^2}{c^2}-\vec{p}^2=m^2 c^2.$$
A photon is somewhat tricky. You cannot fully understand it without quantum field theory. It is described by a massless quantum field with spin 1, the electromagnetic field. An energy-momentum eigenmode of the field is characterized by the three-momentum eigenvalues ##\vec{p}##, and the energy is given by the relation
$$\frac{E^2}{c^2}-\vec{p}^2=0.$$
This explains, why the field is called "massless", because that's a similar relation as for classical particles with an invariant mass of 0.
Of course, one shouldn't call the "invariant mass" "rest mass", because a massless quantum can not be at rest. It always moves with the speed of light wrt. any inertial observer.
)
