I don't understand the book's argument either, and the infinite-well example is everything else than simple. Ironically it is presented to students in QM 1 as if it were the most simple example.
To answer your question we have to look at the Hamiltonian, defined on the Hilbert space ##L^2([0,a])##,
$$\hat{H}=-\frac{1}{2m} \mathrm{d}_x^2.$$
I use natural units with ##\hbar=1##.
The Hamiltonian must be a self-adjoint operator, and this defines the domain, i.e., the function, where it is defined. Obviously the function should be differentiable twice. Now we check hermiticity. Integrating twice by parts you get
$$\int_0^a \mathrm{d} x \psi_1(x)^* \psi_2''(x) = (\psi_1^* \psi_2'-\psi_1^{*\prime} \psi_2)|_0^a + \int_0^a \mathrm{d} x \psi_1'' \psi_2.$$
For the Hamiltonian to be at least hermitean, the non-integral piece must vanish. This doesn't constrain the boundary conditions very much. Indeed possible conditions are (a) ##\psi(0)=\psi(a)=0##, (b) ##\psi(0)=\psi'(a)=0##, (c) ##\psi'(0)=\psi(a)=0##, (d) ##\psi## is a periodic function on entire ##\mathbb{R}##, restricted to the interval ##[0,a]##.
So we need more constraints. One idea is to demand that ##\hat{x}##, defined as ##\hat{x} \psi(x)=x \psi(x)##, should be a self-adjoint operator. That it is Hermitean is trivial, but to be self-adjoint implies that the domain of ##\hat{x}## equals its co-domain. If we now check the above examples of conditions we see that (a) for sure fulfills this constraint, because if ##\psi(0)=\psi(a)=0## then also ##\hat{x} \psi(x)=x \psi(x)## fulfills this condition, and with ##\psi## also ##x \psi## is square integrable over the finite interval ##[0,a]##. So these boundary conditions make at least sense.
Now we have to see, whether also ##\hat{H}## or for this matter ##\mathrm{d}_x^2## is indeed self-adjoint. To that end we evaluate the eigenfunctions of this operator
$$u_{E}''(x)=-\epsilon u_E(x), \quad \epsilon=2 m E.$$
The solutions are
$$u_{E}(x)=A \sin(\sqrt{\epsilon} x)+B \cos(\sqrt{\epsilon} x).$$
Since ##u_{E}(0)=0## we necessarily have ##B=0##. To also fulfill
$$u_{E}(a)=A \sin(\sqrt{\epsilon} a)=0,$$
we must have
$$\sqrt{\epsilon} a=n \pi, \quad n \in \mathbb{Z}.$$
Tha means the ##\epsilon \geq 0##. For ##n=0## the solution becomes identically 0. So it's not a solution either, because eigenfunctions must not be the null vector of the Hilbert space. For ##n>0## the solution with ##-n## leads to the same function up to a factor ##-1##, so it's no new solution. Thus the complete set of eigenvectors are
$$u_{n}(a)=A \sin(\sqrt{\epsilon_n} a), \quad \epsilon_n=\left (\frac{n \pi}{a} \right)^2, \quad n \in \mathbb{N}=\{1,2,3,\ldots \}.$$
Since
$$u_n''(x)=-\epsilon u_n(x)$$
fulfills the boundary conditions and since any function doing so can be written as a Fourier series on this interval,
$$\psi(x)=\sum_{n=1}^{\infty} A_n \sin(n \pi x/a),$$
The domain and the codomain of the Hermitean operator ##\hat{H}## are the same, and thus ##\hat{H}## is self-adjoint on the here considered space. Everything is a consistent description of a particle in the infinite-potential box.
Some puzzle to think about: What about the operator ##\hat{p}=-\mathrm{i} \mathrm{d}_x## (a candidate for a momentum operator)? Is it Hermitean? Is it self-adjoint?