the nucleus has protons arranged in a small volume then why doesn't the nucleus become unstable due to repulsion?
They are bound by the nuclear force, or the residual strong force, which is currently understood to be something of a side affect of the forces that bind quarks into protons in the first place.
In other words, the force that binds protons into protons has residual affects on the "next step" of complexity in particles.