This is one of the reasons I tend not to post too much. It takes too darned long to find good references. I had 30 minutes. The stuff I wanted cost money to access, these are some free relevant ones:
This discusses the discontinuity of phonemic categorization in adults v small children:
The common definition of Non-native means phonemes heard after age 1:
How non-native phoneme recognition is impeded:
See slide 38:
If you look at the first 3 boxes, and if you care to google for more follow up on those terms, go for it.
If you think I'm wrong, that's fine too. I can't devote any more time to this right now.
Biological systems are plastic, but they have inherent limits. I personally believe phoneme categorical perception of previously unheard phonemes in adults to be one of those limits.