I will talk mostly about the stable octet baryons, because excited baryons can have vastly different radial states. The distance between two quarks in a spin zero state is smaller than the distance in a spin one state because of the spin-spin interaction of quarks. This is why the Delta has a larger radius than the nucleon, and the Lambda has a smaller radius. There is a longer range, Coulomb-like, force between quarks that would give a smaller radius to heavier quarks. That is why the Xis have a smaller radius than the nucleon. The spin-spin interaction is not large for charmed and bottom baryons. That is why they tend to have a smaller radius than the nucleon.