In relativity there are a number of different definitions of mass. In particular,
- rest mass =
invariant mass = proper mass which excludes kinetic energy
-
relativistic mass which includes kinetic energy
Most modern physicists mean invariant mass when they say "mass". But there are some people who mean relativistic mass when they say "mass".
A photon has zero invariant mass but non-zero relativistic mass.
The equation
m_{rel} = \frac {m_{inv}} {\sqrt{1 - v^2 / c^2}}
is valid only for objects moving slower than light. For photons both the top and bottom of the fraction would be zero which makes it meaningless.
An equation that is true for all particles, including photons, is
m_{rel}^2 = m_{inv}^2 + p^2 / c^2
where
p is momentum.