The orthogonality of associated Legendre functions can be proved by using the relationship between Legendre polynomials and associated Legendre functions.
They are related by following expression.
P_{l}^{m} (x) = (1-x^2 ) ^{\frac{m}{2}} \frac{d^m}{dx^m} P_l (x)
Note that the Rodrigues' formula for the Legendre polynomials is,
P_l (x) = \frac{1}{2^l l!} \frac{d^l}{dx^l} (x^2 -1 )^l
Now what we want to know is the value 'I' of following integral for arbitrary chosen positive integers l,l'
I = \int_{-1}^{+1} P_l^m (x) P_l'^m (x) dx
By the trichotomy property of real numbers, l and l' must have a following relation.
l < l' or l' < l or l = l'
Without loss of generality, here we assume l \leq l'.
Now expand the integral by following procedure.
I = \int_{-1}^{+1} P_l^m (x) P_l'^m (x) dx
\Leftrightarrow I = \int_{-1}^{+1} (1-x^2)^{\frac{m}{2}} \frac{d^m}{dx^m} P_l(x) (1-x)^2
\Leftrightarrow I = \int_{-1}^{+1} (1-x^2)^m \frac{d^m}{dx^m} P_l (x) \frac{d^m}{dx^m} P_l' (x) dx
\Leftrightarrow I = \int_{-1}^{+1} \left[(1-x^2)^m \frac{d^m}{dx^m} P_l (x) \right] \frac{d}{dx} \left[\frac{d^m-1}{dx^m-1} P_l' (x)\right] dx
Now this is I = \int_{a}^{b} u(x)v'(x) dx form. So, perform an integration by part.
I = \left[( 1-x^2 )^m \frac{d^m}{dx^m} P_l (x) \frac{d^m-1}{dx^m-1} P_l' (x) \right]_{-1}^{+1} - \int_{-1}^{+1} \frac{d}{dx} \left[ (1-x^2 )^m \frac{d^m}{dx^m} P_l (x) \right] \frac{d}{dx} \left[ \frac{d^{m-2}}{dx^{m-2}} P_l' (x) \right] dx
\Leftrightarrow I = - \int_{-1}^{+1} \frac{d}{dx} \left[ ( 1-x^2 )^m \frac{d^m}{dx^m} P_l (x) \right] \frac{d}{dx} \left[ \frac{d^{m-2}}{dx^{m-2}} P_l' (x) \right] dx
Do the integration by parts for 'm' times, then we have,
I = (-1)^m \int_{-1}^{+1} \frac{d^m}{dx^m} \left[ ( 1 -x^2 )^m \frac{d^m}{dx^m} P_l (x) \right] P_l' (x) dx
We know that P_l (x) is l-th degree polynomial. So after do some easy calculation, we find that the a following factor of integrand is l-th degree polynomial.
\frac{d^m}{dx^m} \left[(1-x^2 )^m \frac{d^m}{dx^m} P_l (x) \right]
So, if l' < l, then I = 0 by the orthogonality of Legendre polynomials.
Thus, the remain problem is the case of l'=l and in this case, it is ok, just consider the highest degree term.
The Rodrigues' formula of Legendere polynomial is given by following expression.
P_l (x) = \frac{1}{2^l l!} \frac{d^l}{dx^l} (x^2 -1)^l
From this, we can find that
(Highest degree term of P_l (x)) = \frac{1}{2^l l!} \frac{d^l}{dx^l} x^{2l} = \frac{1}{2^l l!} \frac{(2l)!}{l!} x^l
So, apply this to find the expression of highest degree term of
\frac{d^m}{dx^m} \left[(1-x^2 )^m \frac{d^m}{dx^m} P_l (x) \right]
then we have,
(-1)^m \frac{1}{2^l l!} \frac{(2l)!}{l!} \frac{d^m}{dx^m} \left[ x^2m \frac{d^m}{dx^m} x^l \right]
After do some calculation in big parenthesis, you'll have
(-1)^m \frac{1}{2^l l!} \frac{(2l)!}{l!} \frac{l!}{(l-m)!} \frac{d^m}{dx^m} x^l+m
and do another differentiations,
(-1)^m \frac{1}{2^l l!} \frac{(2l)!}{l!} \frac{l!}{(l-m)!} \frac{(l+m)!}{l!} x^l
But we know that the highest degree term of l-th degree Legendre polynomial is
\frac{1}{2^l l!} \frac{(2l)!}{l!} x^l
So,
(-1)^m \frac{1}{2^l l!} \frac{(2l)!}{l!} \frac{l!}{(l-m)!} \frac{(l+m)!}{l!} x^l
= (-1)^m \frac{(l+m)!}{(l-m)!} P_l (x) + \cdots (low degree terms)
Therefore, remind the first form of our integral.
I = (-1)^m \int_{-1}^{+1} \frac{d^m}{dx^m} \left[ ( 1 -x^2 )^m \frac{d^m}{dx^m} P_l (x) \right] P_l' (x) dx
Now we know that every calculations of terms that independent with highest degree terms are become to 0, by orthogonality of Legendre polynomials.
I=(-1)^{2m} \int_{-1}^{+1} \frac{(l+m)!}{(l-m)!} P_l (x) P_l' (x) dx
And, here we use the orthogonality expression of Legendre polynomials,
\int_{-1}^{+1} P_l (x) P_l' (x) dx = \frac{2}{2l+1} \delta_{ll'}
we will get the following orthogonality expression of the associated Legendre functions.
I = \int_{-1}^{+1} P_{l}^{m} (x) P_{l'}^{m} (x) dx = \frac{2}{2l+1} \frac{(l+m)!}{(l-m)!} \delta_{ll'}
Hence, the proof is complete.
There are another way to prove this by using the associated Legendre differential equation itself as someone mentioned before. That way is more simple and easy to understand. I think this is clever method, but it is too messy
