You know, I've run into this problem too and it has always baffled me. I dig a hole into compacted soil (the hard stuff to dig in), stick a plant of some sort in the hole, including roots and the soil it was potted in, and fully expect there to be dirt at least in excess of the amount of soil added from the pot the plant was in, and can never seem to find it all. Now, a little soil I know just winds up scattered into the surroundings that I just can't scoop up, and that amount will depend on how wide I made my initial pile of dirt, but this never seems to account for the fact that I wind up still needing more soil to fill in around my plant! And when I do have excess soil, the amount of excess soil left seems inversely proportional to the available space for inconspicuously scattering that excess.