Let's create a universe with only two very small masses. Let's say they are bits of dark matter. Let's create a galaxy of these two bits by placing them in orbit around one another at a fixed distance.
Let's introduce a very weak antigravity force, we'll call "dark energy." This force is wa-a-a-a-ay weaker than gravity by many orders of magnitude, though like gravity it works on all scales (since it pushes on all matter). How might it effect our universe model?
Ok, I think this is a good example. I'll repeat your assumptions in a different way, since they will be crucial to the arguments that follow:
1) The "repulsive force" is
very weak on the scale of the system. It will eventually dominate the potential if one goes to large enough radii, but it will be many orders of magnitude larger than the separation between the masses.
2) The potential is
time-independent; that is, the energy density of dark energy at a particular location in space does not change with time. Since we're talking about the cosmological constant, not some other form of dark energy, this is a required assumption.
Since we've established a time-independent potential, we can define an energy for the orbiting mass.
E=-V_g(r)+V_r(r)+T
where V_g is the potential due to gravity (the usual 1/r potential), V_r is the potential of the "repulsive" force, and T is the total kinetic energy of the orbiting body. The radial coordinate is measured from the center of mass of the orbiting bodies. I've pulled the minus sign out of the gravitational potential to make its attractive nature explicit. For the arguments that follow, it won't matter if the repulsive potential increases or decreases with radius, as long as it stays well below the magnitude of the gravitational potential.
Let's start by considering a purely radial orbit. In this case, the kinetic energy is just:
T=\frac{1}{2}mv_r^2
Now, the question is, can we create an orbit that is bound for all time? To answer this, let's just choose some initial condition. We don't have to think very hard on this one, let's just drop the object from some radius, r. That is, v_r=0 at r=r_0. What is the total energy of the object? To answer that, we just set the kinetic energy to zero:
E_{tot}=-V_g(r_0)+V_r(r_0)
A bound orbit has negative total energy and an unbound one has positive. Since we agreed that V_r(r_0) is much less than the gravitational potential, this orbit must be bound. Since the potential is
time-independent, energy conservation must always apply and the orbit must be forever bound.
We can also examine circular orbits. You seem to claim that they would be unstable in this potential, but we can examine this directly. Setting v_r=0, the energy is given by:
E=-\frac{GMm}{r}+V_r(r)+\frac{L^2}{2mr^2}=V_{eff}
A circular orbit is found by finding the minima and maxima of the effective potential as a function of r. The stability is determined by concavity and positive concavity means a stable orbit. Now we could solve for the first and second derivatives of this function and get explicit answers, but it should be easier to just imagine the shape of the function. In a Newtonian potential, if L is small enough, the function should look something like the red curve in this picture:
http://qonos.princeton.edu/nbond/Kepler78.gif
Disregard the blue curve, as it's not relevant here. As you can see, there's a stable circular orbit in the red curve at x=1. The question we want to now ask is whether or not this changes
qualitatively when we add a small potential with positive sign; that is, does this potential change the shape of the curve? If it's very weak, I don't see how it could. You can imagine adding a very small number to every point on that curve, all it will do is shift the minimum slightly.
So what does this mean? Basically, it means that there are stable circular orbits in the presence of a cosmological constant. The dark energy does not forever push the objects apart. All of these arguments fall apart if the dark energy density is
increasing with time because the potential becomes time-dependent and energy is not conserved. This, again, is the Big Rip.
Obviously (to me) this energy/force (regardless of how small) would tend to spread our two bits apart. Why? Because even if it is very weak, a little bit of it is inbetween the two separate but orbitting bits. They are pushed apart (however slightly). Then, since there's more space, more DE get's inbetween and pushes them apart more effectively. then more DE gets inbetween and again pushes them apart even more effectively, then... you get the idea. Now extrapolate this effect as having been happening for about 10 billion years, each orbit being slightly larger than the last.
You can analyze this same problem by summing the effect of an infinite number of infinitesimal forces, but you'll get the same answer. The series must, at some point, converge. If it didn't, energy and/or angular momentum would not be conserved.
However, DE could never be strong enough in this model to separate single gravitational bodies as there is no room to work and insufficient angular momentum to nullify the great strength of gravity (not to mention electro-static attraction). It's only the equalizing force of angular momentum that would allow a condition of balance that can be disturbed by the DE.
If you were to take an orbit in a Newtonian potential and then suddenly introduce a field of dark energy, you would get a perturbation in the orbit for the reasons you give. However, the orbit will quickly reach a new equilibrium that includes the effects of the dark energy (that is, the three forces will come to a balance). If you don't further change your system, it will remain in this equilibrium for all time.
