@_PJ_ :
Very long and complete thought. But I must disagree, as this kind of thinking have all the typical signs of irrational fears. The strongest indicator is beginning this thought process with «Humans are flawed». There are not. They are the results of millions of years of evolution. Whatever characteristic passed the test of time, it's there for a reason. This is true for emotions, curiosity (which you refer to as
being distracted) or choosing to disregard convention, law or logic (it is needed to adapt oneself to the unpredictability of life).
Speaking of predictability, it is not always the best option. Predictability can often direct you to a certain death when something else changes.
When I read your post, I felt like listening to a preacher selling his religion. First, somehow, people are unfit to take decisions. But, fortunately, there is a god called AI that possesses all the answers, at least answers better than the ones people can come up with. Finally, there are those priests, that you refer to as
designers of the systems, that will be the link between the people and this new god, our savior. Because, you know, the common man won't be able to understand this complex AI they so desperately need.
As a person with a scientific mind, I see someone building a driverless car and think to myself: «Cool! That looks like a fun toy!» (That's an emotion). Then curiosity kicks in: «I wonder how it works?» Maybe I'll follow thru, maybe I won't. But if someone starts selling (forcing?) this to me like it is the solution to all my problems (Do I really have problems?) and I should be obliged to have one, well, I don't find this so cool and fun anymore and that makes me feel sad and frustrated. That's another emotion, one that exists specifically to tell me something is wrong. Maybe there's nothing wrong because just like AI would be, this system of emotions developed for the last millions of years is not perfect in every situation. But, along with other tools I possess, I still trust it more than any new, unproven, tool invented by men.
Maybe I'm putting more intentions in what you are saying than you cared for. But to me, this is where this kind of speech is heading and I prefer to tame the wording at its infancy before I wake up with an authority forcing me to buy what, without any surprise, it sells.
So, once more, I don't feel safer with self-driving cars on the road (nor do I feel safer NOT having self-driving cars on the road). Although, I don't mind that some people choose to work on such project. I actually find that stimulating, the only downside being this talk about AI being the solution to all our problems.