There are 3 triangles: the big one, the little one, and the middle one (the one with the 12 deg. angle). You don't need to do anything with the middle one -- the one that you spent most of your efforts on. Your value for the angle that the bottom of the beam makes with the floor is close, but it's slightly more than 2 degrees. Don't round anything until the very end.
What you want is the altitude of the little triangle. Since the angle is given in degrees, make sure that your calculator is in degree mode. To get that, get the angle (alpha) of the big triangle by using the tangent function. tan(alpha) = 10/40 = 1/4, or alpha = arctan(1/4).
The angle, beta, of the small triangle is alpha - 12. Store this value in memory. tan(beta) = h/40, where h is the altitude of the small triangle. Solve for h in this equation.
The nefarious trig student has to be able to fit into a rectangular hole 16" high by 24" wide. Use the angle beta again -- that's why you stored it -- to calculate the altitude of a slightly smaller triangle that is h2 in height and 38' along its base.
Be sure to include units, since you're working with both feet and inches. If you leave off the units, you be more likely to forget what they are, and won't know what you've ended up with.
Good luck!