Radiometry has (possibly) the most horrid system of units in all of science, except for Photometry. "Brightness" is a photometric quantity (how your vision responds) related to the radiometric quantity 'radiance'. Radiance has units of W/sr*m^2 and is the most basic way to describe a source: how much power is emitted (W/m^2) and in what direction that radiation propagates. Radiance is conserved for all optical systems that do not have absorption present.
The fundamental relationship for radiative energy transport takes into account not only the amount of radiated power, but also the relative orientations of the two surfaces (which is why there's steradians floating around)
A candela (Cd) is a radiometric unit of "radiant intensity" has units of W/sr: in what direction the radiated power propagates. "Radiant flux" is a radiometric quantity with units of W: how much energy flows through a surface per unit time.
Edit: photometric units, like brightness and luminance, take into account the fact that your eye does not respond to all wavelengths the same- you are much more sensitive to green than far-red (or far-blue). There's a 'standard curve', the luminosity function, that allows you to convert from radiometric to photometric units, but it's far more difficult to convert a photometric unit to a radiometric unit.
Hope this helps...