OK, so in the Earth frame, ship #1 must travel a total distance of 0.866025 * 10 = 8.66025 light-years before turning around.
From what I said earlier, if you want ship #1 to catch up with ship #2 at the same time that ship #1 turns around to return to Earth (after 10 years in the Earth frame), that means ship #2 must come to rest relative to Earth when it reaches a distance of 8.66025 light-years from Earth. After 6 years ship #2 will be at a distance of 0.866025 * 6 = 5.19615 light years from Earth, so it still has a distance of 8.66025 - 5.19615 = 3.4641 light years to go, so if it traverses that distance at 0.987433c then the time needed will be 3.4641/0.987433 = 3.5082 years, so ship #1 stops after 6 + 3.5082 = 9.5082 years in the Earth frame.
No, your math is wrong here. In ship #1's frame, ship #2 is traveling at (0.987433c - 0.866025c)/(1 - 0.987433*0.866025) = 0.838115c.
First of all, as noted above ship #2 is actually initially moving at 0.838115c in ship #1's frame (before ship #2 comes to rest relative to Earth), for a Lorentz factor of 1.8332. Second, since ship #2 does come to rest relative to Earth at the turning point, ship #2 changes velocities in the frame of ship #1, so that ship #2 is traveling at 0.838115c for some time and then changes to moving at 0.866025c towards ship #1, until eventually they meet.
Perhaps you want to change the thought-experiment so that ship #2 initially accelerates to (0.866025c + 0.866025c)/(1 + 0.866025*0.866025) = 0.989743c relative to the Earth, then comes to rest at the turning point and waits for ship #1 to catch up, in which case in ship #1's frame ship #2 initially moves away at 0.866025c and then turns around and moves back towards ship #1 at 0.866025c until they meet at the turning point?