I did a semi-insane amount of work on this and related problems a while back.
If you take the Hamiltonian of this system, H
H = 1/2*px[1]^2+1/2*py[1]^2+w*(px[1]*y[1]-py[1]*x[1])+
1/2*px[2]^2+1/2*py[2]^2+w*(px[2]*y[2]-py[2]*x[2])+
1/2*px[3]^2+1/2*py[3]^2+w*(px[3]*y[3]-py[3]*x[3])+
1/2*px[4]^2+1/2*py[4]^2+w*(px[4]*y[4]-py[4]*x[4])
-1/((x[1]^2-2*x[1]*x[2]+x[2]^2+y[1]^2-2*y[1]*y[2]+y[2]^2)^(1/2))
-1/((x[1]^2-2*x[1]*x[3]+x[3]^2+y[1]^2-2*y[1]*y[3]+y[3]^2)^(1/2))
-1/((x[1]^2-2*x[1]*x[4]+x[4]^2+y[1]^2-2*y[1]*y[4]+y[4]^2)^(1/2))
-1/((x[2]^2-2*x[2]*x[3]+x[3]^2+y[2]^2-2*y[2]*y[3]+y[3]^2)^(1/2))
-1/((x[2]^2-2*x[2]*x[4]+x[4]^2+y[2]^2-2*y[2]*y[4]+y[4]^2)^(1/2))
-1/((x[3]^2-2*x[3]*x[4]+x[4]^2+y[3]^2-2*y[3]*y[4]+y[4]^2)^(1/2))
-M/(x[1]^2+y[1]^2)^(1/2)-M/(x[2]^2+y[2]^2)^(1/2)
-M/(x[3]^2+y[3]^2)^(1/2)-M/(x[4]^2+y[4]^2)^(1/2)where M is the ratio of the central mass to the 4 orbiting masses, you can compute the linear stability by finding the characteristic polynomial of a 16x16 matrix given by Hamilton's equations
[tex]
\dot{p_i} = \frac{\partial H}{\partial x_i}[/tex]
[tex]
\dot{x_i} = \frac{\partial H}{\partial px_i}[/tex]
(The above eq's need to be duplicated for y) - with the appropriate initial conditions for all the [itex]x_i,y_i,px_i,py_i[/itex] which represent the initial state of the system.
This is obviously something that you don't want to try by hand.
Here w^2 = M + 1/4 + sqrt(2)/2
The characteristic polynomial looks like
1/8192*(256*x^8+512*x^6*M+256*x^6+256*x^6*2^(1/2)+
256*x^4*M^2+640*x^4*M+1408*x^4*M*2^(1/2)+736*x^4+
320*x^4*2^(1/2)+384*x^2*M^2+1152*x^2*M^2*2^(1/2)+
2400*x^2*M+960*x^2*M*2^(1/2)+592*x^2+624*x^2*2^(1/2)+
2736*M^2+864*M^2*2^(1/2)+1272*M+1368*M*2^(1/2)+
361+204*2^(1/2))*(4*x^2+4*M+1+
2*2^(1/2))*(8*x^4+4*x^2*2^(1/2)+
2*x^2+9*2^(1/2)+8*x^2*M+36*M*2^(1/2))*x^2
which appears to have no real or imaginary parts for M = 100, suggesting that the system is (marginally) stable with a central mass. Pertubations should not grow or shrink.
For M=100, the characteristic polynomial is
1/8192*
(256*x^8+51456*x^6+256*x^6*2^(1/2)+2624736*x^4+
141120*x^4*2^(1/2)+4080592*x^2+11616624*x^2*2^(1/2)+
27487561+8777004*2^(1/2))*
(8*x^4+802*x^2+4*x^2*2^(1/2)+3609*2^(1/2))*
(4*x^2+401+2*2^(1/2))*x^2
Without a centeral mass, the system is not stable for n=4.
The numerical simulations for very high n's with no central mass mentioned are interesting - I haven't done those, though I've seen the results.
They appear to be a system that is linearly unstable, but has limit cycle oscillations, which implies a non-linear sort of "stability".
[add]
ps - adding the central mass terms to the Hamiltonian was a bit of an afterthought, I may have to review how I did this :-(.