I'm surprised there is so much confusion about this question.
A little bit of context is needed. If two photons are moving in opposite directions, I assume we are supposed to assume that the total momentum is zero, and that if they are moving in the same direction, that the total momentum is equal to twice the momentum of a single photon.
This assumption is true in SR, for instance. Since that's where the term 'rest mass' is defined, I assume that's the case you are considering.
So, making this clarification, it should be quite clear that for the system where the photons are moving in the same direction, E^2 - |p|^2 = 0 for each individual photon, and therefore the rest mass (invariant mass) of the system , which in geometric units is simply
\sqrt{(2E)^2 - (2|p|)^2} = 2 \sqrt{E^2 - |p|^2} = 0
because the total energy of the system is twice the energy of each photon, and the magnitude of the total momentum of the system is also twice that of an individual photon.
Similarly, when the photons are moving in the opposite direction, the total momentum of the system is zero, so the mass of the system is 2E (in geometric units), or 2E/c^2 (in standard units), because
\sqrt{(2E)^2 - 0} = 2E
This sort of problem can be found in several textbooks, including Taylor & Wheeler's "Spacetime physics.