To greatly oversimplicate things, it boils down to how the repulsive force from protons competes with the attractive strong nuclear force, with a dash of quantum mechanical rules sprinkled in. Or, rather, dumped in by the gallon since this is absolutely a quantum scale system that can't be analyzed classically except to approximate things to various degrees.
As Paul Colby said, you can look at various models, such as the nuclear shell model, to find out more info.