I didn't show the formula or how to get it, since this appeared to be a multiple choice question in a timed test. Once you realize that each layer must be a triangle number, and only one of the choices is a triangle number you are done.
But assume that the problem was to find the number of balls in the base for a large number of total balls. Now you have to drag out the heavy artillery. If we call the function which gives you the total number of balls given the length of a side T, we have:
T(1) = 1
T(n) = T(n-1) + n(n+1)/2
How do you solve the recurrence? In theory, you guess. But you have a lot of experience to go on. The answer will be a cubic equation in n, will have a divisor of 6, and will produce only whole numbers. The series we have to fit is 1,4,10,20,35...
(n)(n+1)(n+2)/6 = 1, 4, 10, 20...
Looks good to me. ;-)
So T(n) = (n)(n+1)(n+2)/6
For 120 balls,
120 = n(n+1)(n+2)/6
720 = n(n+1)(n+2) You can solve the cubic equation, reduce it to a quadratic, or notice that 8x9x10 = 720 and solve for n by inspection.
Substitute in n(n+1)/2 and get 36 as the correct answer for the original problem.