a) First, we want to compute the number of spheres $S_k$ in the $k$ layer (where $1\le k\le n$), counting from the top. Each layer is a triangular grid of spheres with $k$ rows, and we may use the difference equation:
$$S_{j}-S_{j-1}=j$$ where $$S_{1}=1$$ and $$1\le j\le k$$
The homogeneous solution is:
$$h_j=c_1$$
and we should then expect to find a particular solution of the form:
$$p_j=Aj^2+Bj$$
Substituting this particular solution into the difference equation, we find:
$$\left(Aj^2+Bj \right)-\left(A(j-1)^2+B(j-1) \right)=j$$
$$2Aj-(A-B)=j-0$$
Equating coefficients, we find:
$$2A=1\,\therefore\,A=\frac{1}{2}$$
$$A-B=0\,\therefore\,B=A=\frac{1}{2}$$
Hence:
$$p_j=\frac{1}{2}j^2+\frac{1}{2}j=\frac{j(j+1)}{2}$$
Thus, the general solution is:
$$S_{j}=h_j+p_j=c_1+\frac{j(j+1)}{2}$$
Now we may determine the parameter:
$$S_{1}=c_1+\frac{1(1+1)}{2}=c_1+1=1\,\therefore\,c_1=0$$
Thus, the solution satisfying the given conditions is:
$$S_{j}=\frac{j(j+1)}{2}$$
This number is referred to as the $n$th triangular number, and may be written as a binomial coefficient:
$$t_n=\frac{n(n+1)}{2}=\frac{(n+1)!}{2!((n+1)-2)!}={n+1 \choose 2}$$
Now, we want to find the sum of the spheres in all the layers, thus we may use the difference equation:
$$T_{k}-T_{k-1}=\frac{k(k+1)}{2}$$ where $$T_1=1$$ and $$1\le k\le n$$
The homogeneous solution is:
$$h_k=c_1$$
and we should expect a particular solution of the form:
$$p_k=Ak^3+Bk^2+Ck$$
Substituting this particular solution into the difference equation, we find:
$$\left(Ak^3+Bk^2+Ck \right)-\left(A(k-1)^3+B(k-1)^2+C(k-1) \right)=\frac{1}{2}k^2+\frac{1}{2}k$$
$$3Ak^2+(-3A+2B)k+(A-B+C)=\frac{1}{2}k^2+\frac{1}{2}k+0$$
Equating coefficients, we find:
$$3A=\frac{1}{2}\,\therefore\,A=\frac{1}{6}$$
$$-3A+2B=\frac{1}{2}\,\therefore\,B=\frac{1}{2}$$
$$A-B+C=0\,\therefore\,C=\frac{1}{3}$$
And so we have:
$$p_k=\frac{1}{6}k^3+\frac{1}{2}k^2+\frac{1}{3}k= \frac{k^3+3k^2+2k}{6}= \frac{k(k+1)(k+2)}{6}$$
Thus, the general solution is:
$$T_k=h_k+p_k=c_1+\frac{k(k+1)(k+2)}{6}$$
Now we may determine the parameter:
$$T_1=c_1+\frac{1(1+1)(1+2)}{6}=c_1+1=1\,\therefore\,c_1=0$$
Thus, the solution satisfying the given conditions is:
$$T_k=\frac{k(k+1)(k+2)}{6}$$
And so, we may state:
$$N=T_n=\frac{n(n+1)(n+2)}{6}=\frac{(n+2)!}{3!((n+2)-3)!}={n+2 \choose 3}$$
This number is referred to as the $n$th tetrahedral number, or triangular pyramidal number.
b) To find the height of the stack, consider connecting the centers of the 4 spheres at each vertex with a line segment, each of length $(n-1)d$.
Now, to find the height $h_T$ of the resulting tetrahedron, we may drop a vertical line segment from the apex to the base and label its length $h_T$. Next draw a line segment from the apex down one of its faces bisecting the face, and we find its length $\ell$ from $$\sin\left(60^{\circ} \right)=\frac{\ell}{(n-1)d}\,\therefore\,\ell=\frac{\sqrt{3}}{2}(n-1)d$$.
Now, observing that the vertical line $h_T$ intersects the base in the middle (the point equidistant from the 3 vertices), we can find the length of the line segment $b$ joining $h_T$ and $\ell$ by dividing the equilateral base into 3 congruent isosceles triangles each of which has an area $$\frac{1}{3}$$ that of the equilateral base:
$$\frac{1}{2}b(n-1)d=\frac{1}{3}\cdot\frac{1}{2}\sin\left(60^{\circ} \right)((n-1)d)^2$$
$$ b=\frac{1}{3}\cdot\frac{\sqrt{3}}{2}(n-1)d=\frac{1}{2\sqrt{3}}(n-1)d$$
Now we have a right triangle whose legs are $h_T$ and $b$ and whose hypotenuse is $\ell$, thus from the Pythagorean theorem, we obtain:
$$h_T=\sqrt{\ell^2-b^2}=\sqrt{\left(\frac{\sqrt{3}}{2}(n-1)d \right)^2-\left(\frac{1}{2\sqrt{3}}(n-1)d \right)^2}$$
$$h_T=(n-1)d\sqrt{\frac{3}{4}-\frac{1}{12}}=(n-1)d\sqrt{\frac{2}{3}}$$
Now, observing that we need to add 1 more spherical diameter to account for the radii below and above the tetrahedron, we then may state:
$$h=h_T+d=(n-1)d\sqrt{\frac{2}{3}}+d=d\left((n-1)\sqrt{\frac{2}{3}}+1 \right)$$
