This is not a complete solution to your problem, but perhaps this will get you started:
J = Sum[Subscript[y, m]Log[Subscript[P, m]] +
(1 - Subscript[y, m])Log[1 - Subscript[P, m]], {m, 1, M}]-
\[Lambda]/K Sum[Sum[(Subscript[P, m]Subscript[P, mPrime])^2,
{mPrime, unknown, unknown}], {m, 1, M + CapN}];
Solve[D[J, Subscript[P, m]] == 0, Subscript[P, m]]
You have to provide more information or figure out on your own whether K is a constant in one place and a function on mPrime in another or somehow find a way to bring your idea of summing over selected elements of a set in line with Mathematica's idea of Sum working with an index variable over a sequential list of integral values. Adjust that inner Sum[] and {mPrime, unknown, unknown} accordingly. It is possible to toss out Sum and use other methods, but I'm guessing from your question that you might not be up to figuring that out.
One other polite comment. I do realize that many people have come to believe that they must, or at least should be able to, "desktop publish" everything they do, inside and outside Mathematica, including subscripts, superscripts, primes, choosing case that matches the notation from a paper or personal choice, etc. In my experience that usually seems to make the task about twice as hard to do. Maybe you will have better luck. You may find in a variety of ways and places in Mathematica that support for this is uneven and mixed and difficult to diagnose when something doesn't work as expected, above and beyond the usual level of Mathematica being cranky and difficult to diagnose why something isn't working. I hope it works out for you.
