I suspect it happens like it does in other fields. You might research a problem, find a paper and have some questions and so you contact the author explaining your background , interests and current research and the questions you have about their paper and they confess that they too are interested in this same problem and so the collaboration begins.
Of course, you have to develop some trust and contribute in "equal" amounts otherwise it will fall apart. Many of these collaborations go over years where the mathematicians meet every so often in a conference, exchange email and papers and then something magical happens and they have a breakthrough.
Often collaborations happen between mentor and student where trust has been developed while the student was successfully guided to a PhD. Other times it might be a trust developed among fellow grad students. However it happens, you must trust that the person won't run off with your ideas is important and that you don't do the same.
Look at the case of Andrew Wiles who worked silently on Fermat's theorem until finally he had a proof. However, it had a flaw that needed to be patched and so he struck up a collaboration with one of his former students and they completed the proof.
https://en.wikipedia.org/wiki/Andrew_Wiles
In contrast, look at Einstein's case, he presented his incomplete work on General Relativity to an audience of physicists and mathematicians and caught the interest of Hilbert who then tried to complete the work before Einstein. However, Einstein was able to turn to Marcel Grassman, a fellow graduate student friend who helped him past his difficulty.
more on the dispute:
https://en.wikipedia.org/wiki/Relativity_priority_dispute
Bottomline is that academia can be a tough place to work where everyone is vying for their time in the sun and hope they don't get burned in a premature publish and that's why trust in collaboration is important.
