Let's have a look at a particle in 1 dimension in a box with (a) periodic boundary conditions (that's equivalent for a particle on a circle) and (b) with rigid boundary conditions (i.e., a particle in a flat potential with very high walls; simplified by a "box potential", ##V(x)=0## for ##x \in [-L/2,L/2]## and ##V(x) \rightarrow \infty## for ##|x|>L/2##) and look for momentum eigenstates. For simplicity I use natural units with ##\hbar=1##.
First of all one should define the momentum operator. It's by definition the generator of spatial translations and thus in the position representation is given by the operator
$$\hat{p}=-\mathrm{i} \partial_x.$$
That's not enough to know whether this operator makes sense as the representative of an observable. For that purpuse it should be self-adjoint. I'm not using the rigorous way to maximally extend this operator on Hilbert space but use the usual physicists' arguments.
Let's start with case (a) periodic boundary conditions. This means we work on the representation of the separable Hilbert space ##\mathcal{H}=\mathrm{L}^2([-L/2,L/2])## of periodic functions, i.e., ##\psi(-L/2)=\psi(L/2)##. The first step to see, whether the momentum operator makes sense, is to check whether it's Hermitean. That means for any functions in its domain we should have
$$\int_{-L/2}^{L/2} \mathrm{d} x \psi_1^*(x) \hat{p} \psi_2(x) = \int_{-L/2}^{L/2} \mathrm{d} x [\hat{p} \psi_1^*(x)]\psi_2(x).$$
It is very easy to see that this is really true by integrating by parts.
Further for any wave function also ##\hat{p} \psi(x)## should also be in this function space, and that's the case, because the derivative (if existent) of a periodic function is again periodic. So indeed for this case ##\hat{p}## is a very good candidate for a self-adjoint operator.
If it is a self-adjoint operator, there should be a complete set of eigenstates with real eigenvalues. That also easy to check. The eigenstate equation reads
$$\hat{p}u_p(x)=-\mathrm{i} \partial_x u_p(x) \stackrel{!}{=} p u_p(x) \; \Rightarrow \; u_p(x)=A \exp(\mathrm{i} p x), \quad A=\text{const}.$$
Next we have to fulfill the periodic boundary conditions,
$$u_p(-L/2)=u_p(L/2) \;\ Rightarrow \; \exp(\mathrm{i} p l)=1 \; \Rightarrow \; p=\frac{2 \pi}{L} n, \quad n \in \mathbb{\Z}.$$
To normalize the state we calculate
$$\int_{-L/2}^{L/2} |u_p(x)|^2=|A|^2 L \stackrel{!}{=}1 \; \Rightarrow \; A=\frac{1}{\sqrt{L}}.$$
Now obviously the set of eigenfunctions
$$u_n(x)=\frac{1}{\sqrt{L}} \exp(2 \pi \mathrm{i} x/L)$$
is a complete set of orthnormal states, because you can express any periodic function in terms of its Fourier series
$$\psi(x)=\sum_{n=-\infty}^{\infty} \psi_n u_n(x) \; \Leftrightarrow \; \psi_n=\langle u_n|\psi \rangle = \frac{1}{\sqrt{L}} \int_{-L/2}^{L/2} \exp(-2 \pi x/L) \psi(x).$$
Also ##\hat{p} u_n(x)## is obviously again in the considered Hilbert space. So for the periodic boundary conditions we have a well-defined self-adjoint momentum operator. Note that this can be made rigorous, using both the conventional and the modern rigged-Hilbert space approach. See, e.g.,
http://arxiv.org/abs/quant-ph/0103153
Now for the case of "rigid" boundary conditions. Now we have the realization of the Hilbert space as ##\mathrm{L}^2([-L/2,L/2])## with boundary conditions ##\psi(-L/2=\psi(L/2)=0##. Now already the first condition doesn't work out, because in general if ##\psi(x)## is in the domain of ##\hat{p}##, ##\hat{p} \psi(x)## does not fulfill the boundary conditions. On the other hand, for an ##\mathrm{L}^2## function you can alter the function's values on any countable set of points in the interval ##[-L/2,L/2]##, because it's defined only up to Lebesgue null functions. The Hermitezity check also works out in the same way as in the previous case.
So the real killer argument is the evaluation of the eigenfunctions. It turns out that these are the parity even and parity odd functions
$$u^{(+)}_n(x)=A \cos(p_n^{(+)} x), \quad p_n^{(+)}=\frac{(2 n+1) \pi}{2L}, \quad n \in \mathbb{N}_0=\{0,1,2,\ldots \},$$
and
$$u^{(-)}_n(x)=A \sin(p_n^{(-} x), \quad p_n^{(-)}=\frac{2 n \pi}{2L}, \quad n \in \mathbb{N}=\{1,2,\ldots \}.$$
Here you see that it doesn't work out, because ##\hat{p} u^{(\pm)}## does not fulfill the rigid boundary conditions, and thus indeed ##\hat{p}## is not extendable from a Hermitean to a self-adjoint operator.
However, the ##u_n^{(\pm})## are a complete set of orthogonal functions on this Hilbert space. The reason is also simple to see, because obviously ##\hat{H}=\hat{p}^2/(2m)## is extendable to a self-adjoint operator with the ##u_n^{(\pm)}## as a complete set of eigenfunctions. The eigenvalues of the Hamiltonian are ##E_n^{(\pm)}=(p_n^{(\pm)})^2/(2m)##.
