I'm not sure this is what you meant, but I ended up doing the following.
P1 = incoming mass 4 momentum = (E1),p1,0,0)
P2 = created particle = (E2,p1,-E,0)
P3 = photon = (E,0,E,0)
I then equate: P2=P1 - P3
I squared it and rearranged, and plugged in for E1 = 5/3m, which yielded M =...