How Safe Are Self-Driving Cars After the First Fatal Accident?

[OmCheeto]

Big deal.
A friend of mine said I drive like Mario Andretti!

Was his name "Karl"?
If so, I know him.

my Karl: Bus rider. Not used to being in a car.

 

[BillTre]

Was his name "Karl"?
If so, I know him.

my Karl: Bus rider. Not used to being in a car.

Mani, guy from India.
Enjoyed my driving.

 

[berkeman]

I heard on the news report this morning driving to work that the Uber car was not in self-driving mode at the time of the crash, but I haven't been able to find any reference to that now that I'm at work on my PC. Has anybody else seen anything about this? It was CBS news radio...

 

[Vanadium 50]

I heard on the news report this morning driving to work that the Uber car was not in self-driving mode at the time of the crash

If that came from Uber, that's the same company who blamed (and fired) the safety drivers for running red lights in San Francisco, when in fact it was their software.

 

[nsaspook]

I heard on the news report this morning driving to work that the Uber car was not in self-driving mode at the time of the crash, but I haven't been able to find any reference to that now that I'm at work on my PC. Has anybody else seen anything about this? It was CBS news radio...

I've seen nothing about it but is anyone really surprised that the 'safety' driver was not 100% on visual task? I'm pretty shocked by the amount of abuse this person has been getting online and in the media about their past history and employment by Uber. We are good drivers when we’re vigilant. But we’re terrible at being vigilant.
http://journals.sagepub.com/doi/pdf/10.1080/17470214808416738

The General Problem. The deterioration in human performance resulting from adverse working conditions has naturally been one of the most widely studied of all psychological problems. Amongst other possibilities, the stress arising from an unusual environment may be due either to physico-chemical abnormalities in the surroundings or to an undue prolongation of the task itself. This paper is concerned with the latter form of stress, as it has been found to occur in one particular type of visual situation; a later publication will more fully discuss the implications of these and other visual and auditory experiments (Nlackworth, 1948)

 

[nsaspook]

If the problem is the result a LiDAR malfunction remember why this might have happened to Uber.

https://medium.com/waymo/a-note-on-our-lawsuit-against-otto-and-uber-86f4f98902a1

One of the most powerful parts of our self-driving technology is our custom-built LiDAR — or “Light Detection and Ranging.” LiDAR works by bouncing millions of laser beams off surrounding objects and measuring how long it takes for the light to reflect, painting a 3D picture of the world. LiDAR is critical to detecting and measuring the shape, speed and movement of objects like cyclists, vehicles and pedestrians.

Hundreds of Waymo engineers have spent thousands of hours, and our company has invested millions of dollars to design a highly specialized and unique LiDAR system. Waymo engineers have driven down the cost of LiDAR dramatically even as we’ve improved the quality and reliability of its performance. The configuration and specifications of our LiDAR sensors are unique to Waymo. Misappropriating this technology is akin to stealing a secret recipe from a beverage company.

https://www.theregister.co.uk/2018/02/09/waymo_uber_settlement/

We have reached an agreement with Uber that we believe will protect Waymo’s intellectual property now and into the future. We are committed to working with Uber to make sure that each company develops its own technology. This includes an agreement to ensure that any Waymo confidential information is not being incorporated in Uber Advanced Technologies Group hardware and software. We have always believed competition should be fueled by innovation in the labs and on the roads and we look forward to bringing fully self-driving cars to the world.

https://www.reuters.com/article/us-...er-self-driving-incident-safely-idUSKBN1H1006

LAS VEGAS (Reuters) - The head of Alphabet Inc’s autonomous driving unit, Waymo, said on Saturday that the company’s technology would have safely handled the situation confronting an Uber self-driving vehicle last week when it struck a pedestrian, killing her.

 

[HyperTechno]

My condolences to the family who lost one of their important members.

Or simply an unavoidable accident (for example the pedestrian tripped over something and fell on the road)?

She didn't tripped. I saw the dash cam footage. She came out from shadows. The car didn't even try to brake as it seemed, so clearly the car didn't identify the lady. Driver was looking down somewhere not watching ahead so he missed it too. But the way it happened, I mean it happened very quickly. As I saw in the video, it seemed that even if the driver saw her and immediately braked, he would not have been able to stop the car before it hit her because the car was traveling too fast for such a short distance stop. BUT, that impact would've been lower which might have saved her life with severe injuries. But the driver was not attentive and car didn't see her so no braking happened before the car hit the lady it seemed.

 

[nsaspook]

https://www.theverge.com/2018/3/27/17168606/nvidia-suspends-self-driving-test-uber

Uber has been using Nvidia’s self-driving technology in its autonomous test cars for a while, though the companies only just started to talk about it earlier this year. Uber has said it would use Nvidia’s tech in its eventual self-driving fleets of Volvos as well as the company’s autonomous trucks. But Uber has also halted its AV testing in all the cities in which it operates, and the governor of Arizona suspended the company from testing its self-driving cars in the state “indefinitely.”

https://www.engadget.com/2018/03/27/nvidia-self-driving-virtual-simulation/

 

[berkeman]

Well, that was quick. Uber has reached a settlement with the victim's family:

http://www.kron4.com/news/national/...f-woman-killed-by-self-driving-car/1087372323

 

[Grands]

@Dr. Courtney
It happened recently https://www.independent.co.uk/life-...rs-crash-driver-death-elon-musk-a8277306.html.

https://o.aolcdn.com/images/dims?qu...ture=454afdfd0f7cf9abaafd10432fb3c7184bb13a5b

 

[berkeman]

@Dr. Courtney
It happened recently https://www.independent.co.uk/life-...rs-crash-driver-death-elon-musk-a8277306.html.

https://o.aolcdn.com/images/dims?quality=100&image_uri=http://o.aolcdn.com/hss/storage/midas/77e2032fbc30866afd8c3a6fb9563874/206244950/tesla-model-x-crash-dean-smith-twitter-ed.jpg&client=cbc79c14efcebee57402&signature=454afdfd0f7cf9abaafd10432fb3c7184bb13a5b

That's kind of off-topic for this thread. This thread is about the failure of the self-driving car to see the pedestrian that it hit at night. The links you posted are for a completely different accident (that happened about 10 miles from me) where the driver hit a center divider. You can start a separate thread about that accident if you like -- it's not clear that the car was in self-driving mode, IIRC.

 

[dipole]

One question I think is what do we mean when we ask how safe self-driving cars are.

I think deaths per mile isn't a very meaningful measure. For example, suppose in the normal use case of day to day driving, a SDC will have an accident at a rate that is 50% less than a human, and let's define accident as any incident in which people or property are damaged. This certainly seems "safe".

However, suppose in the case of rare events, such as a pedestrian's trajectory intersects that of the cars, the SDC has an accident rate that approaches 90% - meaning, in 90% of such rare events, the car will have an accident. And let's assume for humans that this rate is low, comparable to the overall rate of accidents.

If these events are rare enough, then the total accidents per unit mile of an SDC may well be less than that of human drivers. However, if SDCs have known flaws where they will likely fail, such as consistently plowing into pedestrians, then in those particular instances they should be considered extremely unsafe.

Does a SDC need to outperform humans in all possible cases to be safe? If, on occasion, a SDC will plow down pedestrians (including children), but in the most common scenarios perform better than humans, resulting in overall fewer deaths per mile, are they safe or not?

 

[rootone]

My younger brother once claimed that I drove like a paranoid schizophrenic

I am surprised that a paranoid schizophrenic would have any chance of passing a driving test.

 

[Vanadium 50]

I am surprised that a paranoid schizophrenic would have any chance of passing a driving test.

Why? I don't think "inability to parallel park" is listed as a symptom in DSM-5.

 

[Windadct]

While - a few pages ago I noted that there are 15 pedestrian deaths every day - I was also thinking today that; many cars today have a "breaking override" functionality as part of their safety package where the vehicle applies the brakes on it's own, without driver interaction. This exact technology IS applicable to self-driving - and we are not discussing how many lives have (possibly) been saved by that... I have seen no one promote the data regarding lives saved by this.

Furthermore we have no way of comparing this case to if it had been a human driver...

Taking one (tragic) failure and speculating on the issues based on the video, which we get to stop and analyze, Monday morning quarterdeck style and none of us have direct access to all of the details is counter productive - we are letting our personal opinion and emotion overrule good critical analysis. It falls into populist mindset that results in flat Earth and anti-vaxer movements.

My opinion is based largely on industrial robotics - a MAJOR benefit is Safety - honestly.. humans are AWFUL when it comes to process consistency and reliability - you can not re-train them effectively - they are fatigued, they are distracted, they just do not care... just because YOU think you are better then the average driver... IMO I really do not believe it...

And again - I am NOT for preventing you from driving, mostly because if you are enjoying it your are engaged, but 99% of the driving is just a utilitarian pursuit - get me from point A to point B...

 

[rootone]

Why? I don't think "inability to parallel park" is listed as a symptom in DSM-5.

Yes but attempts to avoid collisions with imaginary objects would be.

 

[mfb]

Does a SDC need to outperform humans in all possible cases to be safe?

Requiring this would kill thousands or tens of thousands of people.
Imagine the opposite direction: If driverless cars would be the default, would we switch to human drivers if they demonstrate to be better at recognizing polar bears on wet roads, but perform worse in all other cases?

 

[nsaspook]

Requiring this would kill thousands or tens of thousands of people.
Imagine the opposite direction: If driverless cars would be the default, would we switch to human drivers if they demonstrate to be better at recognizing polar bears on wet roads, but perform worse in all other cases?

Yes, safe is not perfection but what is safe in these cases?
A 'safe' human driver can hit and kill a pedestrian under exactly this conditions and not be charged with a crime or even given a traffic ticket.

https://www.usatoday.com/story/opin...-needs-set-safety-standards-column/460114002/

The risk of premature deployment is well-understood: Public welfare is harmed by the deployment of unsafe technology. However, delayed deployment would deprive society of the improved safety benefits this technology promises. Without guidance, companies will inevitably err both in being too aggressive and deploying prematurely, or in being too cautious and continuing to privately develop the technology even once it is safe enough to benefit the public. Neither scenario is in the public interest.

For the driver safety aspect.
https://www.tesla.com/en_EU/blog/update-last-week’s-accident?redirect=no

In the moments before the collision, which occurred at 9:27 a.m. on Friday, March 23rd, Autopilot was engaged with the adaptive cruise control follow-distance set to minimum. The driver had received several visual and one audible hands-on warning earlier in the drive and the driver’s hands were not detected on the wheel for six seconds prior to the collision. The driver had about five seconds and 150 meters of unobstructed view of the concrete divider with the crushed crash attenuator, but the vehicle logs show that no action was taken.
...
Tesla Autopilot does not prevent all accidents – such a standard would be impossible – but it makes them much less likely to occur. It unequivocally makes the world safer for the vehicle occupants, pedestrians and cyclists.

 

[nsaspook]

I heard on the news report this morning driving to work that the Uber car was not in self-driving mode at the time of the crash, but I haven't been able to find any reference to that now that I'm at work on my PC. Has anybody else seen anything about this? It was CBS news radio...

Maybe this is what you heard. It's logical that the embedded Volvo XC90’s driver-assistance system would be disabled to have full control during testing.
https://www.bloomberg.com/news/arti...-suv-s-standard-safety-system-before-fatality

“We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Aptiv PLC spokesman Zach Peterson told Bloomberg. “The Volvo XC90’s standard advanced driver-assistance system ‘has nothing to do’ with the Uber test vehicle’s autonomous driving system.”
...
Mobileye, which produces the sensor chips in the safety systems supplied to Aptiv PLC, told Bloomberg it tested the software Monday following the crash by watching the footage of the accident. The company said the software “was able to detect Herzberg one second before impact in its internal tests” despite the video’s poor visual quality.

 

[CWatters]

How do people feel about autonomous cars being deliberately allowed to hit things? I'm not talking about people but things like overhanging tree branches. In parts of the UK there are many single track roads with passing places. It's quite common for cars to brush over hanging branches on such roads. Is this something that's going to be easy for AI to accommodate? What happens when two autonomous car meet on such a road? Anyone testing this or is it all being done in American cities?

 

[nsaspook]

How do people feel about autonomous cars being deliberately allowed to hit things? I'm not talking about people but things like overhanging tree branches. In parts of the UK there are many single track roads with passing places. It's quite common for cars to brush over hanging branches on such roads. Is this something that's going to be easy for AI to accommodate? What happens when two autonomous car meet on such a road? Anyone testing this or is it all being done in American cities?

That will be tricky for the classifier part of object recognition to integrate in the driving prediction plan like 'seeing' the flying bird or floating trash bag in front of the car. I'd be more worried about how quickly human drivers and pedestrians will bully self-driving cars in traffic to make them give way in just about any situation.

https://spectrum.ieee.org/transport...e-big-problem-with-selfdriving-cars-is-people

It’s not hard to see how this could lead to real contempt for cars with level-4 and level-5 autonomy. It will come from pedestrians and human drivers in urban areas. And people will not be shy about expressing that contempt. In private conversations with me, at least one manufacturer is afraid that human drivers will bully self-driving cars operating with level-2 autonomy, so the engineers are taking care that their level-3 test cars look the same as conventional models.

Bullying can go both ways, of course. The flip side of socially clueless autonomous cars is the owners of such cars taking the opportunity to be antisocial themselves.

One person walking like this could slow traffic to human walking speed because would you risk a possible human collision if you designed the automation to always be safe and never take risks people do everyday while driving in traffic.

 

[CWatters]

+1

In such situations it might be something simple such as making eye contact with the person or a small gesture that makes the difference between you having to waiting or pass.

 

[nsaspook]

+1

In such situations it might be something simple such as making eye contact with the person or a small gesture that makes the difference between you having to waiting or pass.

I have my own solution for pedestrian bullies.

 

[nsaspook]

I can think of some easy and devious ways to fool driving AI systems. Tricking self-driving cars could become a national pastime, like wearing a cycling jersey with a full sized STOP sign on the back.

https://arxiv.org/pdf/1802.08195.pdf

Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich. However, it is still an open question whether humans are prone to similar mistakes. Here, we create the first adversarial examples designed to fool humans, by leveraging recent techniques that transfer adversarial examples from computer vision models with known parameters and architecture to other models with unknown parameters and architecture, and by modifying models to more closely match the initial processing of the human visual system. We find that adversarial examples that strongly transfer across computer vision models influence the classifications made by time-limited human observers.

Now that researchers have found trivial ways to hack deep learned visions system they are turning their attention to humans.

 

[mfb]

You can mess with human drivers in many ways as well. Many of these ways are illegal, some ways specific to autonomous cars might become illegal, but overall this is not a big issue today, I don’t see how it would become one in the future.

 

[nsaspook]

You can mess with human drivers in many ways as well. Many of these ways are illegal, some ways specific to autonomous cars might become illegal, but overall this is not a big issue today, I don’t see how it would become one in the future.

There is a inherent danger in messing with human drivers because they can also act irrationally.
I hope so but in a culture where automation has eliminated driving and most low skilled jobs young people will find something to do to kill time instead of the 'Tide Pod' challenge.

 

[mfb]

Passengers in driverless vehicles can act irrationally as well. But I don't think that is what stops people from harming traffic deliberately.

 

[Grands]

That's kind of off-topic for this thread. This thread is about the failure of the self-driving car to see the pedestrian that it hit at night. The links you posted are for a completely different accident (that happened about 10 miles from me) where the driver hit a center divider. You can start a separate thread about that accident if you like -- it's not clear that the car was in self-driving mode, IIRC.

The autopilot was active https://www.theguardian.com/technology/2018/mar/31/tesla-car-crash-autopilot-mountain-view, Tesla confirmed, this is the reason I posted here.

 

[russ_watters]

You can mess with human drivers in many ways as well. Many of these ways are illegal, some ways specific to autonomous cars might become illegal, but overall this is not a big issue today, I don’t see how it would become one in the future.

Perhaps because they are computers people will want to hack them?

 

[mfb]

And there are people who want to hack websites. That doesn't stop them from existing, and taking over more and more functions - buying and booking things online, online banking, ...
The danger of typing Robert'); DROP table students;-- didn't exist when everything was done on paper - but that still didn't stop the conversion to computer databases. People found exploits like SQL injection, and other people wrote patches to fix them.

If STOP sign cycling jerseys become a trend and the cars struggle with them, someone will write a patch to better distinguish between cycling jerseys and actual stop signs.