Thanks Drakkith and onomatomanic.
To quantify the ability of our solar system to catch massive objects:
Assume that a massive, slow-moving object (10km/s far away) approaches our solar system. Assume that its mass is small compared to the mass of sun (the reason will become clear later). How can we capture it? Gravitational interaction with a planet. It has to dump enough energy to fall below escape velocity - and it has to do so in a single interaction, as two significant interactions with planets in a single pass through the solar system are extremely unlikely.
The best geometry is a head-on approach to a massive, fast-moving planet, with a very near miss: With the approximation that all objects are point-masses, the planet will be shot in the opposite direction, getting a velocity kick of twice the relative velocity. Real planets have a finite size, which can limit the maximal momentum transfer to lower values, but I will neglect this issue here.
We need massive, fast-moving planets close to the sun... I will begin with Jupiter here and consider Mercury afterwards:
Jupiter orbital velocity 13km/s
Escape velocity (solar system) at its distance: sqrt(2)*13km/s = 18.38km/s
Velocity of incoming object at Jupiter orbit: sqrt(10^2+18.38^2) km/s = 20.93km/s
Relative velocity: 20.92km/s+13km/s = 33.92km/s.
=> Jupiter velocity change 2*33.92km/s = 67.85km/s
Required velocity change of incoming object: (20.93-18.38)km/s=2.54km/s.
=> maximal mass of incoming object: 67.85/2.54 = 26.7 Jupiter masses = 0.026 solar masses.
Mercury: Orbital velocity 47.87km/s, maximal mass of incoming object 317 mercury masses = 0.055 Jupiter masses.
As you can see, Jupiter's mass dominates the results - even with Earth in a mercury orbit, 317 Earth masses would be about one Jupiter mass (and not 26.7).
This gives 26.7 Jupiter masses = 0.026 solar masses as an upper limit for any reasonable capturing process in the solar system.
What happens if we take the finite size of the objects into account? Objects with 26.7 Jupiter masses are brown dwarfs, with a size similar to Jupiter.
Escape velocity scales with sqrt(M/r), at twice the Jupiter radius (closest possible flyby without touching) this corresponds to 60km/s*sqrt(26.7/2)=220km/s. Based on an initial relative velocity of 33.92km/s, the velocity at closest approach is sqrt(220^2+33.92^2)km/s=222.6km/s.
Calculate the (minimal) eccentricity:
##e=\sqrt{1+\frac{2\epsilon h^2}{\mu^2}}## with ε=1/2 (33.92km/s)^2, h=4*(jupiter radius)*(222.6km/s) and μ=G*26.7*(jupiter mass)
=> e=1.175
This gives a maximal deflection of 2.04 or 117° - in other words, only ~85% of the maximal velocity change can be used and the maximal mass is even lower. And a minimal separation of 2 Jupiter radii is not possible anyway - the brown dwarf is extremely dense, its Roche limit for Jupiter will be significantly larger.
I would expect ~15 Jupiter masses as a more reasonable number.