Virous, rather than try to unpack your notation, which I (like others) am having trouble understanding, perhaps it would help to show how I would work this problem, using standard 4-vector notation. Each ball is described by a 4-momentum vector, which is just its rest mass ##m## times its 4-velocity ##u^a## (where the index ##a## denotes the coordinate components of the 4-vector, i.e., the components of ##u^a## are ##u^t##, ##u^x##, ##u^y##, and ##u^z##). The ##t## component of 4-momentum in a given frame gives the object's energy in that frame, and the ##x##, ##y##, and ##z## components give the components of its momentum. So checking conservation of momentum before and after just amounts to checking the appropriate components of the 4-momentum before and after. Since the invariant mass ##m## of each ball is the same throughout, we can divide through by it and just work with the 4-velocity vectors.
I won't give a lot of explanation of what follows; the formalism I'm using is described in most relativity textbooks. I'll just work through it quickly so you can see how it goes.
In the unprimed frame, the two 4-velocity vectors before the collision are (giving the ##t##, ##x##, and ##y## components; all ##z## components are zero):
$$
u_1^a = \left( \frac{1}{\sqrt{1 - a^2 - b^2}}, \frac{a}{\sqrt{1 - a^2 - b^2}}, \frac{b}{\sqrt{1 - a^2 - b^2}} \right)
$$
$$
u_2^a = \left( \frac{1}{\sqrt{1 - a^2 - b^2}}, \frac{- a}{\sqrt{1 - a^2 - b^2}}, \frac{- b}{\sqrt{1 - a^2 - b^2}} \right)
$$
After the collision, the two 4-velocity vectors are:
$$
u_1^a = \left( \frac{1}{\sqrt{1 - a^2 - b^2}}, \frac{a}{\sqrt{1 - a^2 - b^2}}, \frac{- b}{\sqrt{1 - a^2 - b^2}} \right)
$$
$$
u_2^a = \left( \frac{1}{\sqrt{1 - a^2 - b^2}}, \frac{- a}{\sqrt{1 - a^2 - b^2}}, \frac{b}{\sqrt{1 - a^2 - b^2}} \right)
$$
Obviously the ##x## and ##y## components of the two vectors cancel both before and after, so momentum is conserved. (Btw, the sum of the two ##t## components is also the same before and after, so energy is conserved as well.)
Now we want to Lorentz transform these 4-vectors to the primed frame, which is moving in the positive ##x## direction at speed ##a##. The nice thing about 4-vectors is that they all Lorentz transform the same; a LT in the ##x## direction only changes the ##t## and ##x## components, and it does so in the same way you're used to from basic relativity physics. That is, we have ##u'^t = \gamma \left( u^t - v u^x \right)##, and ##u'^x = \gamma \left( u^x - v u^t \right)##, where ##v## is the relative velocity (##a## in this case) and ##\gamma = 1 / \sqrt{1 - v^2}## (note that I am using units in which ##c = 1## for simplicity); and ##u^y## and ##u^z## are unchanged.
Now, if you look at the before and after 4-vectors given above, you will notice something: the ##t## and ##x## components are *the same* before and after in the unprimed frame. And since the LT only changes the ##t## and ##x## components (i.e., it doesn't "mix in" any others), the same thing will be true in the primed frame: the ##t'## and ##x'## components will be the same both before and after. So it's obvious that energy and ##x## momentum will be conserved in the primed frame if they are conserved in the unprimed frame (which of course they are). And since the ##y## components are unchanged by the LT, ##y## momentum will also be conserved in the primed frame.
So we really don't even need to work through the details of transforming into the primed frame; but I'll do it anyway for completeness. Applying the LT, in the primed frame we have these two 4-velocity vectors before the collision (where I have refrained from simplifying the expressions so you can see how they arise from the general LT formula above):
$$
u_1^a = \left( \frac{1 - a^2}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{a - a}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{b}{\sqrt{1 - a^2 - b^2}} \right)
$$
$$
u_2^a = \left( \frac{1 + a^2}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{- a - a}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{- b}{\sqrt{1 - a^2 - b^2}} \right)
$$
Simplifying now, we get
$$
u_1^a = \left( \frac{\sqrt{1 - a^2}}{\sqrt{1 - a^2 - b^2}}, 0, \frac{b}{\sqrt{1 - a^2 - b^2}} \right)
$$
$$
u_2^a = \left( \frac{1 + a^2}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{- 2a}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{- b}{\sqrt{1 - a^2 - b^2}} \right)
$$
After the collision, all that changes is that the ##y## components swap, as before, so we have:
$$
u_1^a = \left( \frac{\sqrt{1 - a^2}}{\sqrt{1 - a^2 - b^2}}, 0, \frac{- b}{\sqrt{1 - a^2 - b^2}} \right)
$$
$$
u_2^a = \left( \frac{1 + a^2}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{- 2a}{\sqrt{1 - a^2 - b^2} \sqrt{1 - a^2}}, \frac{b}{\sqrt{1 - a^2 - b^2}} \right)
$$
Again, it's obvious that the ##y## momentum cancels, and that the total ##x## momentum, while it is no longer zero in this frame, is the same both before and after. (And, of course, so is total energy, although it is larger in this frame than in the unprimed frame.)
