In summary, the equations given can be satisfied in three different scenarios: (a,b) = (0,0) with any x, y, and z values, (a,b) = (a,0) with a non-zero and (a,b) = (a,12a2) with a non-zero and specific x, y, and z values. Any other values of (a,b) will not have a solution.
If a = 0, then (x = 0 or y = 0) AND (x = 0 or z = 0) AND (y = 0 or z = 0) so it's easy to see that b = 0. Any triple with at least two of x, y, and z being 0 will satisfy the given equations.
If a is nonzero and anyone of x, y, or z is 0, then it turns out that all three must be 0, from the first three equations. E.g. if x = 0, then 0 = xy = a(x+y) = ay, and since a is non-zero, y must be zero; then we get 0 = yz = a(y+z) = az and since a is non-zero, z too must be zero. So we only get a solution if b is zero, and that solution is (0,0,0).
If a is nonzero and none of the x, y, z are 0, then we get:
xy = a(x+y)
1/a = 1/x + 1/y [dividing both sides by axy]
Likewise, 1/a = 1/x + 1/z and 1/a = 1/y + 1/z. It's easy to see that this gives 1/x = 1/y = 1/z, i.e. x = y = z. x2 = xy = a(x+y) = 2ax, so x = 2a. So we'd get b = 12a2.
Summary:
(a,b) = (0,0) has solutions (0,0,z), (0,y,0), and (x,0,0) for any x, y, and z
(a,b) = (a,0) with a non-zero has the solution (0,0,0)
(a,b) = (a,12a2) with a non-zero has the solution (2a, 2a, 2a)
(a,b) = anything else has no solution