In general, a linear operator is a mapping from one vector space to another. For any vector space, we can consider the special case of linear operators that map the vector space to itself. The starting point for the formalism of QM is that we have a vector space of valid states (or wave functions) for the system and each observable is represented by a linear operator on that vector space. I.e. the operator maps the vector space to itself.
For the infinite square well, the valid wave functions must be zero at the boundary. This means that the momentum operator is not a valid linear operator on this vector space of wave functions. This is because the derivative of a function is not necessarily zero at the end points.
This technicality doesn't stop the infinite square well being used as an interesting example of wave mechanics. If you think about it, a truly infinite potential is unphysical.
I would draw an analogy with the infinite uniformly charged plate in classical EM. The electric field is uniform, hence a particle in that field has infinite potential energy. All of these examples: infinite straight wire, infinite plate, infinite square well are all likely to run into a mathematical/physical problem at some point.