The net charge on the conductor is enough.
That is the 'uniqueness theorem' tha is proved in most EM texts.
You start with the volume integral of phi grad phi, where phi is the difference of two possible potentials for the same rho (so delsquared phi=0.
Then use the divergence theorem.
