There are three kinds of mass: gravitational source mass ([itex]M_S[/itex]), gravitational response mass ([itex]m_t[/itex]), and inertial mass ([itex]m_I[/itex]). The first two are the two kinds of mass that appear in Newton's gravitation law.
[tex]
F=\frac{M_Sm_t}{r^2}[/tex]
The third is the kind of mass that appears in Newton's second law of motion.
[tex]
a=\frac{F}{m_I}[/tex]
As you can see, I have (purposely) omitted G from Newton's gravitational law. However, I believe initiated by the work of Galileo, we eventually developed the tradition of converting gravitational source mass and response mass to inertial mass. However, at least one of the two kinds of gravitational mass must have different units than inertial mass in order for Newton's gravitational law to give units of force. What Galileo discovered was that
[tex]
m_t=c_{tI}m_I[/tex]
where [itex]c_{tI}[/itex] is some universal constant that converts from the units of inertial mass to gravitational response mass. Also, motivated in part by Kepler, Newton suggested that the roles of [itex]M_S[/itex] and [itex]m_t[/itex] should be interchangeable (due to his third law of motion). So
[tex]
M_S=c_{tI}M_I[/tex]
Putting this into Newton's gravitational law makes it a universal law in terms of inertial masses, and one finds a constant of proportionality
[tex]
c_{tI}^2\equiv{}G[/tex]
As has been suggested, G was not able to be determined in Newton's time. What scientists were able to conclude was that G is universal, which is still a big deal. It is a profound statement to say that the amount of stuff is directly proportional to the amound of gravity it produces and also directly proportional to how strongly it responds to gravity. In fact, this is such a profound notion that it eventually led Einstein to propose the gravity isn't even a real force, just a geometrical phenomenon. It should be noted that general relativity could have just as well developed if we never knew the numerical value of G, just as long as we were confident that it was universal.