This is one of the most frequently asked question, and also one of the most important, for beginners in this field. I am not sure of your background in this field. But photochemistry and photophysics are probably best understood as statistical quantum mechanics. This is going to be long, but brace yourself if you want to learn.
First, I have to make sure you understand the quantum scale of the discussion.
Any transition is dominated (most of the time) by kinetics. Anything that is easier to happen, happens more than those that are harder to happen (obviously). We express this in terms of probability, so the former has higher probability for the event to happen while the latter has lower probability for the event to happen. What probability is this talking about? It's talking about how many times an event happens per unit time (rate in which an event happens):
k [event/s]. Most of the time, "event" in the unit of
k is abbreviated and is just shown as s
-1. There is an equation called Fermi's Golden rule shown by the following equation:
k = \frac{2\pi }{\hbar}\left | \left \langle \phi_{i}\left | \mathbf{V} \right | \phi_{f}\right \rangle \right |^{2}\delta \left ( E_{f}-E_{i}-\hbar\omega \right )
This equation shows that the rate of event to occur is dominated by two factors: 1) how strong the interaction V between initial state \phi_{i} and final state \phi_{f} is, and 2) how well the energy between these two is matched, described by a delta function of \delta \left ( E_{f}-E_{i}-\hbar\omega \right ) (energy conservation law). If we are talking about electronic transition between two states, namely \phi_{i} and \phi_{f}, then the energy conservation is easily met by an emission of light corresponding to the energy gap between the two states (\hbar\omega). So what we have to worry here is the former: electronic correlation between the two states (the matrix element).
If a transition is "forbidden" for the two levels we have just mentioned, it means that the matrix element \left \langle \phi_{i}\left | \mathbf{V} \right | \phi_{f}\right \rangle is zero to the first order approximation, making
k = 0 (this transition does not happen). What does "to the first order" means? Well electronic correlation is dominated by whether the atom is in the right place, electronic overlap is existent, and if the spin state of initial and final state is conserved. These parameters are calculated separately and multiplied. If one or more of these parameters are zero, then the whole matrix element is zero, thus "forbidden". However, in reality, initial \phi_{i} and final state \phi_{f} is mixed to some degree due to several second-order and third-order effects (such as spin-orbit coupling, configuration interactions, electron-phonon coupling, crystal field, etc.), and this matrix element may, although small, take a non-zero value. This introduces some "allowedness" to the forbidden transition, hence why sometimes we observe emission from something that we are not supposed to (for example, lanthanides' 4f-4f transition). That means
k is not zero and the transition is observed.
What if we have several electronic levels? For example, if we have three states in the energetic order of \phi_{3}, \phi_{2}, \phi_{1}, how do we know which transition from state \phi_{3} is more often observed? Well we do the above calculation. Which ever has the larger
k value is more likely to happen. So your first question:
How does the electron decides which transition (among the allowed ones) it will make among the many possible? Is that a random, probabilistic choice?
It's probabilistic.
Say we can observe one single atom with three energy levels (experimentally, this is actually possible by use of fluorescence microscopy). Here, we use monochromated light that excites the atom to the third level (\phi_{3}) only. The atom will absorb the light and emit a light whether that may be \phi_{3} to \phi_{2} transition or \phi_{3} to \phi_{1} transition. All of these processes (including absorption of light) are probabilistic in terms of time. So what happens if we continue to excite an atom and look at it? Well the atom actually blinks! How fast does it blink? It depends on how probable absorption and emission are (how large
k) is. Larger it is, faster it blinks. Which transition will the atom blink of?
BOTH! However, one transition may blink more than the other if
k value for \phi_{3} to \phi_{2} and \phi_{3} to \phi_{1} is different.
So now you see why an atom show two different emission.
Now let's move on to the macroscopic scale of the discussion. What happens if we have many of the same atom? Although each atom blinks, because we have so many of them, it seems as if they are constantly and steadily emitting. So when you look at the spectrum, it seems like the atom is emitting constantly from all transitions. So your thought:
Two different atoms in the same sample, both in the same energy state, may choose different transition paths
is right.
But,
However, the various lines are NOT produced by the same atom.
is only right if we are talking about the instant moment an atom emits. If we talk about in the longer time scale (more than several blinks), the same atom is actually producing both emissions. It's simply blinking, each time of different transition. In total, we have so many of these atoms that they look like they are constantly emitting different wavelength of light.
Your first point discusses the situation in which an atom with multiple electronic levels are excited by non-monochromated light (light of many wavelength). The discussion is still the same, only except more complicated because we have to take into account of different absorption and different emission.
Your second point discusses blackbody radiation, which is a thermal radiation. Think about it in terms of statistical thermodynamics combined with statistical quantum mechanics. All you have to do is make numerous energy level of very very small energy gap, and think about the thermaldynamic equilibrium. This numerous energy levels are not electronic levels, but phonon levels. What we've talked until here is about electronic levels (atomic spectra), but in blackbody radiation, we are talking about phonon levels (phonon is a quasiparticle of quantized vibration). This time, the excitation source is not an electromagnetic wave (light or photon), but heat. Putting a blackbody material in an environment of high temperature means more heat energy is given to this blackbody. Due to this heat energy, the blackbody transitions to higher phonon level. Emission occur from the excited phonon levels. How much of the blackbody is located at certain phonon level at a given temperature?
Boltzmann distribution!. Max Planck used Boltzmann distribution to derive the infamous Planck's law that explains spectral irradiance (related to intensity of emission) of a given temperature
T and wavelength
λ:
B(v,T) = \frac{2hc^{2}}{\lambda ^{5}}\frac{1}{e^{\frac{hc}{k_{B}T}\frac{1}{\lambda }}-1}
The e^\frac{hc}{k_{B}T}\frac{1}{\lambda} part is directly from Boltzmann distribution (
hc/λ represents energy). So your second point:
How does the blackbody radiation theory patch together with the theory of line spectra?
We are looking at different transition here. For line spectra, we are looking at emission of light by absorption of light, involving electron levels. For blackbody radiation, we are looking at emission of light by absorption of heat, involving phonon levels.
If we excite a gas with a discharge tube don't we also raise its temperature?
Well yes, but in order to observe visible emission, we need at least about 1000K (730°C, 1340°F)! As Charles Link mentioned, gas of discharge tubes don't emit well. So I wouldn't think too much about it. Besides, such gases are far from ideal blackbodies.There you go. I hope you understand this. You can ask more question if something wasn't clear.