Consider a container with a removable wall in the middle. On the right are many gas molecules moving relatively slowly (cold). On the left are many gas molecules moving relatively quickly (hot). If we remove the wall the hot and cold molecules will begin to collide with each other. The hot molecules, upon collision with their cold counterparts, will transfer some of their speed and momentum and thus cool down. This means, of course, that the cold molecules heat up.
In order for heat to flow from the cold to the hot these collisions would require that the slow molecules move even slower after a collision and the fast molecules move even faster. This is an impossibility in isolated collisions, and a rarity in multi-body collisions.