PeterDonis said:
This is a much more complicated question than it appears to be.
One issue is that the reasoning you are probably using, that the "relativistic mass" of the objects is larger when they are moving at relativistic speeds, is wrong. "Relativistic mass" is not the source of gravity in GR; in fact the term itself is no longer used much because it invites too much confusion. The basic problem with "relativistic mass" as a source of gravity is that it is frame-dependent, and the source of gravity can't be frame-dependent; that source has to be described by something that doesn't depend on your choice of reference frame. In GR, that something is the stress-energy tensor.
A second issue is that we need to be clear about what "gravitational attraction" is, and how we would compare it in the two scenarios you describe. Intuitively, we might think it should be something like "acceleration due to gravity", but the kind of acceleration that term refers to is, like relativistic mass, frame-dependent; in fact we can make it disappear altogether by an appropriate choice of reference frame (a local inertial frame). So again, we need to have some frame-independent way of characterizing the mutual "gravitational attraction" of the two objects. Unfortunately, there isn't a single simple frame-independent mathematical object, like the stress-energy tensor above, that serves that purpose in GR.
We might think we could at least talk about the overall deflection in the paths of the two objects as they pass each other; but this also has issues, since the higher the relative speed of the objects, the shorter the time during which they are close together and can affect each other's motion significantly.
There is one thing we can say, and that is that the total energy of the two-object system, in its center of mass frame, will be larger in the first scenario (relativistic relative speed) than in the second (very small relative speed), assuming that the rest masses of the objects are the same in both scenarios. (This also assumes that there are no other significant masses present, so that spacetime as a whole is asymptotically flat and we can have a well-defined "total energy".) But that doesn't necessarily translate in any simple way to "more gravitational attraction between the objects".
I agree that it's not an easy problem, but there has been at least one paper written about it.
We can imagine a stationary test mass - or a cloud of such masses, and ask what velocity is induced in the mass (or the cloud of masses) by a relativistic flyby.
Olson & Guarino do this in their paper " "Measuring the active gravitational mass of a moving object". They impute the velocity gained during the entire flyby to an "active gravitational mass". They get the rather interesting result that what they term "the active gravitational mass" scales by a factor larger than the relativistic gamma factor, namely ##(1+\beta^2) \gamma##, where ##\beta = v/c \quad \gamma = 1/\sqrt{1-\beta^2}##. In the flat space-time before and after the relativistic flyby, the velocity of the test masses relative to some distant object not affected by the flyby can be unambiguously defined, as can the velocity change due to the flyby.
This is not really too surprising, in light of Tollman's paradox.
As I recall, Olson & Guarino do the analysis for the induced velocities (there are components transverse to the flyby, and components parallel to the flyby) both for GR and a Newtonian flyby, though I can't confirm my recollection as I only have access to the abstract of the paper at this time.
For the case where ##\beta = .5##, ##(1+\beta^2)\gamma## would be a factor of 1.44 increase in the induced velocity of the test mass due to the flyby.
It's still wrong to say that the relativistic mass is what causes the gravity, there is a 2:1 discrepancy.
I find the OP's question a bit vague, as it's not clear that he is imagining that we have one stationary mass and one moving mass, which is the case the analysis by the above authors works out. If the OP was asking about a case where both masses were moving, the answer would probably be different than what they expect. In the situation where one mass is stationary , frame dragging aka "gravitomangetic" effects are not important, while they are important if both masses are moving. If both masses were moving with identical velocites, for instance, covariance would suggest that the easiest thing to do would be to analyze the problem in the rest frame of the two masses, and convert this to the moving frame. Doing an analysis with "forces" would confuse the picture because forces are observer dependent. This could be fixed either by using 4-forces, which are observer independent, or concentrating on the induced velocities, i.e. the kinematics (an analysis in terms of motion, not forces). The later kinematic approach would probably be the cleanest.
