However, I believe this blanket statement is not that far off.
The only reference that comes to mind is a Nova (PBS) program dealing with early childhood language sound processing and neural pathway development. The example given was for a Salish language which had unique phonemes. The bottom line was that children after the age of one year could not learn to speak the language correctly - when they had not been exposed to the sounds previously.
Now all I have to do is to find the references.