I certainly use Stirling's formula a lot, so all my results can only be valid for large N. I agree completely.
One big problem occurs in section 6 when I try to specify what the chemical potential and the density of states actually is for an ideal gas. That's where I switch to an integral with an infinite upper limit although the total energy is finite. I guess that's related to what you point out.
I really only use an infinite upper limit so that I can evaluate the integrals for total energy and particle number.
Until section 6 I only consider a finite set of discrete energy levels, which ought to mean that I can keep the highest energy level below the total energy simply by definition.
I simply shouldn't try to apply the theory to anything real! :)