Thanks for the reply. I understand all the equation you wrote, it is exactly like what I have except \Theta(\theta)\rightarrow \; P_n^m(cos\theta). I just stop type it all and try to put in as little as possible because my experience from the past if I type in step by step, nobody reply because it is too long and people fell asleep!
\mu can only have non trivial solution if \mu=n(n+1)
\frac{\partial^2Y}{\partial \theta^2} + cot\theta \frac{\partial Y}{\partial \theta} + csc^2 \theta \frac{\partial^2Y}{\partial \phi^2} + \mu Y = 0
Y(\theta,\phi) = \Theta(\theta)\Phi(\phi)
\frac{\partial^2Y}{\partial \theta^2} + cot\theta \frac{\partial Y}{\partial \theta} + csc^2 \theta \frac{\partial^2Y}{\partial \phi^2} + \mu Y = \frac{\Theta ''}{\Theta} + cot\theta \frac{\Theta '}{\Theta} + csc^2 \frac{\Phi ''}{\Phi} + \mu = 0
\Rightarrow\; sin^2\theta \frac{\Theta ''}{\Theta} + sin^2\theta cot\theta \frac{\Theta '}{\Theta} + sin^2\theta \mu = -\frac{\Phi ''}{\Phi} = m^2
Therefore:
\Phi''+m^2\Phi = 0 \Rightarrow \Phi(\phi) = e^{jm\phi}
\Theta '' + cot\theta \Theta ' + (\mu-csc^2\theta m^2) = 0
Let s= cos\theta
\Theta '' + cot\theta \Theta ' + (\mu-csc^2\theta m^2) = (1-s^2)\frac{d^2\Theta}{ds^2} - 2s\frac{d\Theta}{ds} + (\mu -\frac{1}{1-s^2}m^2)\Theta = 0
The Last one is Assoiate Legendre equation and the solution is P^m_n(s)=P_n^m(cos\theta).
Y(\theta,\phi) = \Theta(\theta)\Phi(\phi) = e^{jm\phi}P_n^m(cos\theta)
Still that does not answer my question why \mu=n(n+1) except the solution is bounded if \mu=n(n+1) like the book said.