[add]
I've taken a more detailed look at your results:
Interestingly enough, there does seem to be something going on here that I'm not familiar with. The equation that I give below predicts that instability starts when m/M = .019819, or 2m/M \approx .0396
The stability analysis I did was a purely linear one - what sort did the author do in your textbook?
The results that I know about are as follows:
In order for three massive bodies to be stable (as defined by a linear analysis) one must have the following relationship between their masses:27 \, (m_1*m_2 + m_2*m_3 + m_3*m_1) < (m_1 + m_2 + m_3)^2
The best literature reference I have for this is Volume 5 of of "What's Happening in the Mathematical Sciences" by Barry Cipra.
If m_3 is zero, this reduces to the usual relationship (see for instance http://www.physics.montana.edu/faculty/cornish/lagrange.pdf for the usual 2-body case).
Once upon I time I took it upon myself to confirm this result via a rather involved computer analysis based on the Hamiltonian approach. One can find a summary of what's involved in the following post.
http://groups.google.com/group/rec.arts.sf.science/msg/1fd8079db42c1137?
Basically one writes down the Hamiltonian (preferred over the Lagrangian in this case because it gives a first order system of differential equations).
One then can linearize the resulting equations, and by computing the eigienvalues of a 12x12 matrix formed from a series of partial derivatives of the Hamiltonian, one can find the above stability conditions.
Your (Tony's) gravity simulator program should be able to empirically demonstrate the above results - when I was working on this myself, a long time ago, I ran some simulations as a first step to see if the equation above worked. (Another poster, Joseph H, originally gave me the equation, as you can see if you dig back in the history of the news thread above). The simulations I did gave unstable behavior when the equations were violated, and stable behavior when they were satisfied.
Apparently, however, you've already done some work in this area, and gotten different results. Interesting.