As I listed, I was studying French, German, Russian, and Spanish, and my native language is English. There are sounds in all those languages that just don't exist in English and which trip up every beginning student. The German "ch" is a good example. We never make this sound in English and it requires learning from scratch.
There were plenty of sounds in all these languages I wasn't reliably hearing at first, which was true of all the students, but the teachers always police these pretty rigorously; point them out to you and require you to drill them. You may or may not take the fact that people don't pick up on them on their own as confirmation of what you're saying, but to me the important thing is whether you can hear it and learn to reproduce it once it's brought to your attention. The problem is often not that people can't hear it, but that they completely dismiss it's importance. It's not obvious that the failure to mimic a certain sound will stand out like a sore thumb in the ears of native speakers. Someone has to tell you, "The way you're pronouncing that is completely outside anything a native speaker would ever produce."
On that subject, there was a Japanese guy who lived in my building several years back. He told me most English classes in Japan were taught by Japanese teachers who butchered the accent. When he arrived here he could not understand one word anyone said despite the fact he could read and write English extremely well. He had to re-learn the accent from scratch, and, when I met him, he could pronounce "r" and "l" correctly, having learned by immersion. It could well be the first time he heard either properly pronounced it didn't register in his brain at all, and it may not have registered till the tenth or twentieth time, but eventually it registered and he could pronounced them properly. The fact that new phonemes stop registering automatically after one year of age is probably irrelevant to the issue of whether one can become fluent.
I suspect that the Salish sounds you're talking about might be harder than most to distinguish, but this would represent a freak rather than a norm. If you foot the bill I'm certainly willing to accept the challenge of traveling to the Pacific Northwest to see if I can learn to pronounce them to the satisfaction of Natives, hehe.
Opining here: the sounds a person produces are very personal and become mixed up with their identity. I think that's most people's biggest problem in picking up a foreign language: they balk at the accent because to dig in and adopt it would cause a minor identity crisis. Actors, mimics, and children don't have that psychological block.