High compared to what?
To slow down neutrons, it is useful to have targets with a mass similar to the neutron mass:
- If you throw an elastic ball on a wall (~heavy nucleus), it will bounce back with the same energy
- If you throw it on another ball (~light nucleus), both will receive a part of the kinetic energy
- And if you hit a fly with it (~electron), it will just continue to fly in the initial direction
Hydrogen has the best mass. But protium (1 proton, 0 neutrons) can absorb the neutron, and deuterium in large amounts is expensive.
Helium is a gas, which is bad for the required density.
Lithium has some safety issues, and the additional neutron absorption problem (in Li-6)
No idea about beryllium. It is toxic, but you don't want to get in contact with anything in the reactor core anyway.
Boron is fine if you want to absorb neutrons... ;)
Carbon (especially graphite) is cheap and solid within a useful range of temperatures