"Orthogonal", of course, comes from geometry meaning "perpendicular". One property of that is that if two vectors are perpendicular their dot product is 0.
It can be generalized to any "inner product space" with orthogonal defined as "the inner product" (a generalization of dot product in Rn). For spaces of functions, such as "[itex]L_2[a, b][/itex]", the set of all function that are "square integrable" on interval [0, 1], we can define the inner product to be [itex]\int_a^b f(x)g(x) dx[/itex] (or complex conjugate of g for complex valued functions). Two such functions, f and g, are said to be "orthogonal" if [itex]\int_a^b f(x)g(x)dx= 0[/itex].
One can show, for example, that [itex]\int_0^{2\pi} sin(nx)sin(mx)dx= 0[/itex] and [itex]\int_0^{2\pi} cos(nx)cos(mx)dx[/itex], as long as [itex]m\ne n[/itex] and that [itex]\int_0^{2\pi} sin(nx)cos(mx)= 0[/itex] for all m and n. Thus, the set of functions {sin(nx), cos(nx)} for for an "orthogonal set" of functions.