Zero is definitely wrong. X can have at least two elements, e.g. {9, 16}. But can X have infinitely many? I doubt it. If X could have infinitely many, then there would be some natural n such that there are infinitely many Pythagorean triangles that have one non-hypoteneuse side with length equal to n. There's a theorem:
You will get every primitive Pythagorean triple (a,b,c) with a odd and b even by using the formulas a = st, b = (s2 - t2)/2, c = (s2 + t2)/2 where s > t > 1 are chosen to be any odd integers with no common factors.
Note, a primitive Pythagorean triple (PPT for short) is a triple (a,b,c) such that a, b, and c have no common factors (i.e. common to all three simultaneously) and a2 + b2 = c2.
So suppose X contains some element x. Let F(x) be the full set of factors of x, i.e. F(x) = {d in N : d | x}. If (x,y,z) is a Pythagorean triple, then there exists some d in F(x) such that (d,yd/x,zd/x) is a primitive Pythagorean triple. Proof: set d = x/gcd(x,y,z). So to classify the Pythagorean triples involving x (but x not being the hypoteneuse) it suffices to classify all PPTs involving d (with d not in the hypoteneuse) for each d in F(x).
Before we go on, note that if (a,b,c) is a PPT then one of {a,b} is even and the other is odd. Both can't be odd because the sum of two odd squares is not a square by modulo 4 arithmetic. Both can't be even because the sum of two even squares is even, so if a Pythagorean triple results, it can't be primitive. Now if d is odd, there are only finitely many pairs (s,t) such that d = st, thus only finitely many PPT involving d. It's also quite easy to see that if d is even, there are only finitely many pairs (s,t) such that d = (s2 - t2)/2, so there are again only finitely many PPT involving d. Thus, for any x, there are only finitely many Pythagorean triples involving x (again, x not hypoteneuse) and so X cannot be infinite.
Note that this does not prove that |X| cannot be an arbitrarily large finite number, just that X cannot be infinite. The above doesn't do much in terms of telling us how big X can be, it just rules out two extreme cases.
Now I think one would start by looking at 3 distinct positive elements of X, say x, y, and z. Let x + y = Y, x + z = Z. W.l.o.g., let z > y. Now we have Y - y = Z - z. Just to be clear, x, y, z, Y, and Z are squares; x, y, and z are elements of X; Y and Z may not be in X; X is a set, not a number. Let's also define a, b, c, B, and C to be the square roots of x, y, z, Y, and Z respectively. So
Y - y = Z - z
(B - b)(B + b) = (C - c)(C + c)
Recall y < z, so Y < Z, b < c, and B < C. Of course, we also have that Y > y and Z > z, so C > c and B > b. And a, b, c, B, and C are all naturals. We know that B + b < C + c, and it's not hard to check that B - b > C - c. So we have C + c > B + b > B - b > C - c.
I don't know if this is leading anywhere. It looks like a pretty hard problem to me, on the other hand there might be a nice, short solution that will make me look stupid ;).