I don't observe this to be true at all here in Southern California. The many people who have citrus fruit trees not only eat the fruit themselves, they give it to neighbors, or take it to the swap meet and sell it.
When I lived in NH it was a somewhat different story. So many people had apple trees that there was a surfeit of apples when they all came ripe in the fall. It could be hard to find someone who didn't already have enough apples for their needs.
My grandmother had a cherry tree and a plum tree, which were a bit unusual for NH. She had to go to great length to keep people from stealing
So, I think the answer may possibly be that there's a surfeit of fruit where you live.