Silly Questions said:
Why do I not know about other problems with FM? The peril of asymmetrical filtering isn't intuitively obvious to a beginner reading an FM theory book, sorting that question into the category of, "Didn't know to ask." The theory book I read didn't talk about how things might go wrong.
I re-read this para and I see that you are not really aware of what Frequency Modulation is actually doing. Your question is definitely trying to run b before you can walk. (Very risky)
In an arm waving way we can liken FM to wobbling the frequency control knob on a lab oscillator in step with the voltage variations of the input signal. Not wrong - but it misses the essence of what's happening and the consequences of that form of modulation.
Silly Questions said:
If you had a monitor attached to the transmitter which perfectly subtracted the reference frequency and compared it to the output of an FM receiver of the on-air transmission, the difference in outputs between the two would be what,
This passage is actually meaningless. What do you mean by "subtract the reference frequency"? In a wide deviation system, like FM radio the "reference frequency" is no more relevant to the transmitted signal than a frequency that's 1kHz above or below. If, by subtracting, you mean use a narrow filter at the nominal carrier frequency then all that would do is to reduce the transmitted power a bit, each time the carrier sweeps through that 'notch'. Basic FM is fairly tolerant to that sort of nasty. FM is very different from basic AM because in AM, the carrier is an essential part of the signal (a reference) and it's there all the time, carrying its Amplitude Envelope. As I have said before, often (for simple sinusoidal modulation at a particular frequency) there is no frequency component at the notional carrier at all.
It might help if you realize that Frequency Modulation is a sub-set of Phase Modulation that's not coherent - i.e. there is no reference carrier involved. However FM is demodulated, the demodulator output has no definite zero or origin.
OTOH, phase modulation can be demodulated coherently and there is an implied reference carrier frequency and phase from beginning to end.
Remember:
Phase is the Integral of Frequency (Imagine the angle of a wheel turning clockwise and anticlockwise along with a modulating signal.)
And Frequency is the time differential (rate of change) of Phase.
You can measure the Frequency without any knowledge of a reference phase and, as we know, when we do an indefinite Integration, the answer involves an arbitrary constant. For FM radio we don't care about that constant.