DrStupid said:
As Nugatory already explained, Ep it is not the potential energy of m but of the system consisting of m and M. The total energy of this system is
E = \left( {m + M} \right) \cdot c^2 - G \cdot \frac{{m \cdot M}}{r}
As it decreases when the objects get closer, your idea sounds reasonable so far. However, to get a negative total energy the objects would need to get so close that the Newtonian gravitational potential is no longer an acceptable approximation. It needs to be replaced by a relativistic potential energy V (which doesn’t need to be limited to gravity). If I understand correctly then you are asking whether
E = \left( {m + M} \right) \cdot c^2 + V < 0
is possible in general relativity or not. My intuitive answer is No. I guess that the system will collapse to a black hole (which cannot release energy anymore, exept by Hawking radiatiation) before its total energy drops to zero or even below. But I have no idea how to prove it.
I think a way to make this (nearly) rigorous in GR and get away from any issues of the conventionality of potential energy (as well as the fact that potential energy is not generally definable in GR, certainly not in this case for an initial condition that is dynamically unstable thus cannot be consistently made part of a stationary spacetime) is conceptually as follows. Each step is non-trivial, but within the state of the art:
1) Define a series of asymptotically flat spacetimes evolved from series of initial cauchy surfaces as follows: each cauchy 'melds' 2 SC geometries of chosen mass parameters, such that they are some initially stationery distance apart per some plausible harmonic coordinate system (to use ADM evolution). This is highly non-trivial, but is what is done e.g. when the final inspiral stages of BH are simulated from an arbitrary starting point. The series uses smaller and smaller distances. Note that there is no really consistent past continuation leading to such a state, but the BH inspiral experts work around this (it is easy to reason that any prior state includes gravitational radiation that we are not including in our initial cauchy surface).
2) Compute the ADM mass of each such (incomplete - unspecified before the initial cauchy surface) manifold. For given chosen mass parameters, this should decrease as the initial distance decreases. This ADM mass is physically the total gravitational mass of the system measured at spatial infinity. It will include the effects, per full GR, of the mass discount from binding energy.
I am not aware of anyone actually doing such computations, but agree the likely outcome is that you would be dealing with infinitesimally separated apparent horizons without total ADM mass reaching zero.
Note, that extant BH inspiral calculations that I know of, are not relevant because they simply follow one history from some initial state, and the GW emitted includes only some of the initial potential energy. Part of it becomes kinetic energy and and angualar momentum 'captured' by the final BH.
This would be pretty close to a proof that you can never get negative total energy (when it is definable) by virtue of objects being very close together, in the sense that a mixture of Newtonian potential and SR might seem to imply.