My English are pretty poor! I will try again to explain.
- From the conservation of momentum you can find the velocity [itex] V_A[/itex] of M, in terms of the velocity v of m.
- From the conservation of energy for the mass M, from A to B, you can find the velocity [itex]V_B[/itex] in terms of [itex]V_A[/itex], i.e. in terms of v.
- Now demanding [itex]V_B=0[/itex] in order the pendulum barely swings you can calulate v.
Does it makes any sence?