sawer said:
As you answered my other thread, there mustn't be any sign there. Because it is "magnitude" of the gravitational force. I think there is no unit vector there. It must be displacement vector.
I think that you are getting confused here. To be clear, I will use ##\vec{x}## to denote a vector and denote its magnitude ##|\vec{x}|## and I use ##\hat{x}## to denote a unit vector defined such that ##|\hat{x}|=1##.
Now, when you write a vector in coordinates you write it as a sum of some basis vectors, like ##\vec{a} = a_x \hat{x} + a_y \hat{y} + a_z \hat{z}##, where the basis vectors are defined at each point as the unit vector in the direction of increasing coordinate. So ##\hat{x}## is the unit vector which points in the direction that the x coordinate increases (other coordinates held constant). Therefore in spherical coordinates the ##\hat{r}## vector points in the direction of increasing r coordinate, which is outwards, away from the origin.
Up to this point all of this is purely math, with no physics content. Now, suppose we have a central particle of mass ##M## located at the origin and a test particle of mass ##m## located at coordinates ##(r,\theta,\phi)## in a spherical coordinate system. From Newton's law of gravitation we know that the gravitational force is proportional to both ##M## and ##m## and inversely proportional to ##r^2##. We also know that the gravitational force is attractive, so the force on the test particle points inwards, towards the origin which is the opposite direction as ##\hat{r}##.
So, if we write the gravitational force on the test particle as ##\vec{F}=-G\frac{Mm}{r^2} \hat{r}## then we get a force that points in the correct direction, i.e. it attracts towards the origin. If we omit the negative sign then we would erroneously get ##\vec{F}## pointing in the same direction as ##\hat{r}## which is away from the origin, i.e. a repulsive force on the test mass.
The dot product doesn't even enter in yet. There is a negative in the gravitational force law itself, regardless of what you might calculate afterwards. You have to start with the correct force law, and only then can you begin to calculate correct quantities based on it. Is that clear?