First, this should not be posted under "Learning Materials". This area is for tutorials, not questions, so I am moving it to the homework section.
Second, you have
[tex]\int_a^b\left(M(x)+ h(x)\right)^2 dx= \int_a^b M^2(x)+ h^2(x) dx[/tex]
and "differentiate with respect to x". Since those are definite integrals, they are numbers, constants, not functions of x and "differentiating with respect to x" just gives 0= 0.