Gravitational Lagrangian PE term

[Dale]

I was just doing a simple 2-body Newtonian gravitation problem. The force on each mass is:

$$f=\frac{G m_1 m_2}{r^2}$$

and the integral of the force wrt r is:

$$\int \! f \, dr = -\frac{G m_1 m_2}{r}$$

So, since there are two forces in the system, one on each object, I had assumed that there would be two potential energy terms so the total potential energy would be twice the above integral, or:

$$U = -2\frac{G m_1 m_2}{r}$$

but I checked my work using Newtonian mechanics it turns out that it gives the wrong equation of motion and the correct potential energy is only one times the integral.

So, my question is, can anyone explain how I should have known to only include one times the energy even though there were two objects, and how I can generalize to N-body problems.