Just off the top of my head, I'll say it may have to do with the physical properties of clay and sand, and a little to do with the chemical properties of clay also. I'm not an expert in this, so maybe someone can provide better input, but I have been messing with plants (and therefore with soil too) since I was 8. I've tried mixing sand with clay soil many times, because it seems at first glance it should be an improvement, but clay mixed with sand does seem to form a sort of temporary, weak concrete (until it becomes wet again), is it not much better than the clay alone is. In fact, clay baked at very high temperatures is one of the ingredients of Portland cement, the key ingredient of conventional concrete. Clay and sand are also the main components of many conventional brick recipes. I think what goes on is that clay particles surround and coat every relatively-large sand particle, thereby separating the sand particles from each other and retaining a tight physical fit, so that air and water still can't move through well. I think temporary weak the clay also forms weak chemical bonds with itself and other particles as the clay dries, almost like a weak glue. Organic particles that make up a loam are shaped differently and are mostly much larger than clay particles, can let moisture flow through much better, can chemically react with things better, and attract/harbor microorganisms - anything from bacteria to worms, which further help air and water move through the soil.
The ratio of the soil components would be very important too. For example, I imagine a clay-loam mix would suddenly grow plants much better once the loam content reached a certain percentage (depending on the exact properties of that type of loam), because at some point, many loam particles would be touching each other, providing a 3-D web through which water, air and nutrients could flow, and roots could penetrate easier.