This is why the field of solid state physics is different than the field of atomic physics. As Phil Anderson has said "More Is Different"!
The fact that it is a solid means that there are strong bonds that hold the atoms and molecules in place. So there is already strong coupling between the atoms that make up the solid. It means that when one part of the solid is agitated (as in the supply of heat), then it is possible (but not always) that this agitation can be transferred.
Metals are good conductors of heat due to two reasons (naive picture): (i) they are usually crystalline solids, so the transfer of the vibrational energy is not "haphazard", unlike amorphous or polycrystaline solids; (ii) the conduction electrons are not only very mobile, and thus, can transfer the heat energy easily, but they also have a rather low specific heat capacity, meaning that they heat up very quickly and easily, and give up heat the same way when they reach cooler regions of the metal.
In solids, vibrational energy (something that is not available in isolated atoms) plays a huge role in their properties. This is what you are missing in your scenario, because you are focusing on free particles that can only transfer energy via collision.
Zz.