Physicists assume that observables are represented by self-adjoint ('Hermitian') operators because they guarantee that the corresponding eigenvalues will be always non-negative. The theory was aparently structured in this way because, I think, this might have been somewhat obvious for them at the time. I mean, it is well-known in linear algebra that this is true. Now, there isn't in fact anything that says that there could not be an observable which is not self-adjoint. In fact, an interesing field began in the past decade or so called PT-symmetric quantum theory, where P stands for parity and T for time-reversal. I don't know much about it but it seems interesting. Apparently, if the Hamiltonian (which is an observable) is PT-symmetric (which in general will not be self-adjoint) then the eigenvalues are also guaranteed to be non-negative.
So here's the answer to your question.
Of course there are other operators which are not self-adjoint as was already pointed out, but there is a class of operators, namely those which are PT-symmetric, that seems to be also physically relevant. I suggest you look up in the repository arxiv.org, there are some introductory papers to the subject.