An orthogonal operator basis would just be a basis for various states, associated with whatever operator you're talking about, which is orthogonal. Remember that a basis for a vector space is a set of vectors and any arbitrary vector in the vector space can be written as a linear combination of the vectors in the basis set. If a basis is orthogonal, it means that the inner product of any two of the basis vectors equals zero. A lot of time we use an orthonormal basis. This means that in addition to being orthogonal (inner product = 0) the basis vectors each have a magnitude equalling one. In quantum mechanics each operator has a set of basis vectors associated with it. Often quantum states are written in the position basis or the momentum basis...but you can have a spin basis and other kinds of bases if you wish. If you'd like more information on vector spaces and bases and things like that the subject dealing with those topics is called Linear Algebra.
