okay said:
so if Newtons law (F=G*M1*M2/R^2) expressing the phenomenon in this impossible condition how can he be sure that in this special experiment conditions, force going to be proportional to the masses and reversely proportional to the distance square.
In Newton's time, Kepler's laws were ad-hoc laws that explained the motion of the planets. Newton's laws of motion dictates that because the planets must be under the influence of some kind of force. To give it a name, let's call it solar gravity for now. Why call it "solar gravity"? The planets obviously orbit the Sun. The Sun must be doing something, i.e. exerting a force, on the planets to make this happen. What are some other characteristics of this "solar gravity"?
The mass of a planet is not involved in Kepler's laws. The acceleration that "solar gravity" induces on a planet is solely a function of the distance between the planet and the sun. This means that the "solar gravity" force on a planet must be proportional to the mass of the planet.
There is no reason to think space has a preferred direction or a preferred plane. If solar gravity also has no preferred direction or plane, then the influence of solar gravity must fall off with distance inversely proportional to the surface area of the sphere with radius equal to the distance in question. This means solar gravity is an inverse square law! So now we have, the solar gravitational force acting on a planet p as F_p \sim M_p/r_p^2.
This expression and Newton's laws of motion generate Kepler's laws. Validation! Newton was able to take Kepler's ad-hoc laws to something much deeper. It's still ad-hoc, but much deeper. Newton wasn't done. The Moon orbits the Earth, and other planets were known to have satellites as well. The Moon's orbit about the Earth and the orbits of the moons of Jupiter and Saturn fit nicely into this scheme. However, the proportionality constant is different for the Moon's orbit about the Earth compared to the Earth's orbit around the Sun.
It's not too much of a leap to say that the difference in proportionality constants results from the mass of the central body. After all, the force has to go both ways (Newton's third law). So now we have F_{1\to2} \sim M_1M_2/r_{1\to2}^2 as a much more universal gravitational relationship. This works not only for the planets about the suns but for various moons around different planets.
The theory looks very nice and simple. Nice and simple theories often die horrible deaths when confronted with experimental evidence. Not this one. It wasn't until a nearly a couple of hundred years later that experimental evidence hinted something might be a bit wrong. in the precession of Mercury. Early last century, Einstein showed show how to resolve the problem with general relativity. For the most part, the difference between what Newton's law of gravity says a body will do and who general relativity says a body will do is very, very small.