I don't really get the first part of your question, but solids are held together by bonds between atoms. There are several different kinds of bonds. For crystals like salt, which is composed of the positive ion Sodium and the negative ion chlorine, the solid is held together by the Coulomb force. In the other extreme of a metal, the metals essentially share a "sea" of electrons which can go between the atoms (individually). When electrons away from an atom, the ion becomes positive and has an attraction to valence electrons. A molecular solid (like ice) is an intermediate where there are individual molecules containing covalent bonds. Each atom has an oxidation number, and the bonds are made to fill the valence shells of both with the shared electrons. These molecules are held together by by intermolecular forces like the van der waals force caused by interaction of spontaneous dipole moments of molecules. A covalent solid is a solid made of covalent bonds connecting the atoms (no individual molecules).
As for the stability and structure of these systems, you can synthesize something and it will eventually minimize its energy to find the ideal configuration. However, this doesn't have to be an absolute minima. There is something called being meta stable where the system is at a local minima, so if you somehow get the systems there, it will stay there even if you perturb it slightly. You can see this by looking at phoning modes (lattice vibrations) around the configuration.
The thing you said about atoms having some sort of infinity of space is very misleading. Each atom is a quantum system and likewise has a wave functions. When you put atoms together you get a many body wave function. However, in a lot of these systems the wave function is localized to the site it sits on. In a metal, the positive ion cores can be approximated as stationary and the quasiparticles move around like they are free (can write eigenstates in terms of plane waves). I say quasiparticles because if there are interactions there properties could differ that those of electrons (the charge is the same but they can have a different effective mass).
