Sonuvagun, there does seem to be a physiological basis for it - an actual missing piece of the brain.
"In nine of ten tone-deaf people, the superior
arcuate fasciculus in the right hemisphere could not be detected, suggesting a disconnection between the posterior superior temporal gyrus and the posterior inferior frontal gyrus."
https://en.wikipedia.org/wiki/Tone_deafness
"The
arcuate fasciculus (Latin,
curved bundle) is a bundle of axons that forms part of the
superior longitudinal fasciculus. The arcuate bidirectionally connects caudal
temporal cortex and inferior
parietal cortex to locations in the
frontal lobe."
https://en.wikipedia.org/wiki/Arcuate_fasciculus
Essentially a bundle of axons connecting Broca's area with Wernicke's area.