In fact, any material will dope the semiconductor if they have energy level(s) which lies within the bandgap. But the issue is with the degree of doping. Apart from the well known dopants like (Boron and Phosphorus) the dopant energy level is much below the CB of above the VB. Thus complete ionization does not occur, requiring much greater dose of dopants.
There are also other issues. A group II or group VI impurity differs much in atomic radius than group IV Si/Ge, creating compressive or expansive stresses. This will generate defects and/or alter the carrier mobility degrading the quality of the semiconductor.
Even for legitimate dopants, fab labs have to deal with many issues. Like the activation energy is very high for In. Al is easily oxidized. Diffusion of Ga in SiO2 is high requiring thicker oxide mask. P is 10 times diffusible in Si than Sb and As making the doping profile difficult to control. Even As is sometimes discarded for being poisonous.
Thus, when it comes to select a dopant, its selecting one which ionizes well, produces little defects, can be controlled well, has less poison hazard.