Potential energy is never "absolute" but is relative. In particular, potential energy due to gravity is always dependent upon height relative to some base height. For what height is the potential energy 0?
Gravitational Field(Gf) is defined as the negative gradient of the Gravitational Potential. There has to be another object there in space to actually have a gravitational force attached to the field.
Potentials themselves have no direction they are scalar quantities.
A Gravitational Field should always have its field lines pointing toward the object creating the field for ease of calculations because gravity is an attractive force.