You said it yourself: The solutions to the differential equations are ellipses. More precisely, the solutions to the Newtonian two-body problem are in the form of conic sections. Ellipses are one kind of conic section.
A better question: The solar system after all has more than two bodies in it, the whole solar system is orbiting the galaxy, and Newton's law of gravitation is only approximately correct. So why are orbits elliptical in shape?
The answer is that they aren't exactly elliptical in shape. They are instead approximately elliptical. The Sun is by far the dominant mass in the solar system. Planets perturb the orbits of each other from a purely elliptical shape. The primary effects are to slowly change the orientation and shape of the ellipse over time. General relativity similarly can be viewed, at least in our solar system, as a perturbative force. As far as the solar system orbiting the galaxy and the motion of the galaxy itself, those effects are so tiny as to be unmeasurable, at least as far as planetary orbits are concerned.