As you know, one of the fundamental laws of nature is the one that says objects move in straight lines at constant velocity unless acted on by a force.
If you apply a force to the object, it will gradually change its path and/or velocity. The higher the force, the more sudden a change. For the path to change in a fashion that produces a perfect, sharp bend, like a corner in a square, the force would have to be infinitely large - in any other case (which means, in all cases in reality as there is no such thing as an infinitely large force) the change will be gradual. This removes as a possibility all the shapes that are not rounded.
Now, say you want a rectangle with rounded corners. The trajectory would be such that there is a bend, then a long straight stretch, then another bend and so on. This would require whathever force is acting on the object to periodicaly turn itself on and off, or at least change direction so that at times it acts in line with the object's trajectory, at times at an angle.
Gravity doesn't do that. It's always "on", and there is always a component of the force perpendicular to the motion*. This necessitates a trajectory that doesn't have straight bits*.
(*unless falling exactly towards the centre of the central mass, but then it's always straight)
Furthermore, due to the way the force of gravity scales with distance (##1/R^2##) , any curving of the trajectory will be such that it is more gradual the farther away the orbiting object is from the central body. This reduces the possible shapes of orbits to "conic sections" - this includes a circle, an ellipse, a parabola and a hyperbola. This fact is known as Kepler's first law of orbital motion, observationally derived by Kepler, and later mathematically derived by Newton from his laws of motion and the law of gravitation.
In general, this behaviour, as well as e.g., the motion being planar, is a property of objects affected by "central forces".
It is worth noting that orbits get more complicated, and more diverging from the ideal conic sections, as you add more gravitating bodies that act to perturb and deflect the trajectories.
You can read more about keplerian orbits here:
http://en.wikipedia.org/wiki/Kepler_orbit
and about properties of central forces here:
http://en.wikipedia.org/wiki/Classical_central-force_problem