Safety and the increasing automation of cars

[Guineafowl]

The recent 737 tragedies have shown us, I think, a few pointers about automation of planes. Of course, investigation is ongoing but my understanding is that the input from a faulty angle-of-attack sensor and an automated system which the pilots couldn’t switch off contributed.

Two worries here - lack of multiple sensors to allow a ‘majority vote’, and no manual override.

Leaving aside driverless prototypes, cars are increasingly going this way - lane departure, auto-brake, stop-start... We’re relying on electronic components placed in harsh environments (vibration, damp, heat cycles) to implement these features. A faulty lane departure could jerk you into oncoming traffic, auto brake could jam on and cause a pile-up, stop-start could wind you across a T-junction (if you’re in first gear and the sensor says you’re in neutral).

1. Are we convinced there’s enough redundancy and resilience built into cars?
2. Do we need a simple ‘all-kill’ switch to allow total driver control, say in the event of a global CAN bus glitch?
3. Will there be a time when we need to put phones, etc into flight-safe mode before setting off?

 

[jrmichler]

I have concerns that are even more basic. I have owned and/or driven several cars where the manufacturer could not even program the cruise control correctly. My current truck has a computer controlled gas gauge that does not correctly report fuel level. This does not give me confidence about their ability to program an autonomous vehicle.

A number of deaths have occurred because people were not able to open windows or doors to escape a submerged or burning vehicle. If the manufacturers cannot build windows and doors that can be opened in an emergency, how can we trust them to build an automated vehicle?

What happens when an oncoming vehicle splashes up a big glob of slush onto the sensors?

 

[scottdave]

Add to those 'harsh conditions' the fact that any maintenance on these things may go neglected. How many of us drive around for awhile with a Check-engine light on? Take it in, and find out it's going to be $500 to replace the sensor...

There is a history of vehicles which would accelerate for unknown reasons.

These will be good discussions for Engineering Ethics courses as well.

 

[anorlunda]

I would like to add to the question. "What should the standard be?"

Let's assume that advocates of driverless cars suggest that accident deaths could be reduced 80% if all vehicles were driverless. But the remaining 20% of deaths were directly attributable to malfunctions or inadequacies of the cars.

But advocates tend to exaggerate. So let's say total deaths are cut by only 40% but the deaths that do occur are the machine's fault.

I suspect that the public would find that to be totally unacceptable. So, what should the standard be? Should we judge based on cold net gain/loss calculations, or does it matter whether the human or the machine is at fault?

 

[sysprog]

I am against a vehicle being operated on the public highway without its operator being at risk of his or her own human calamity just as I am when I'm in or on a vehicle on the public highway.

 

[Orodruin]

But advocates tend to exaggerate. So let's say total deaths are cut by only 40% but the deaths that do occur are the machine's fault.

I suspect that the public would find that to be totally unacceptable. So, what should the standard be? Should we judge based on cold net gain/loss calculations, or does it matter whether the human or the machine is at fault?

The public, honestly, is not fit to judge these things because it tends to judge based on anecdotal level evidence. It does not help you that 5 other lives were spared by automation if your relative was killed in an accident where automation was to blame, and so the uproar starts.

On a societal level, it is clearly better to have 1 person killed by automation than having three killed by human error.

 

[sysprog]

The public, honestly, is not fit to judge these things because it tends to judge based on anecdotal level evidence. It does not help you that 5 other lives were spared by automation if your relative was killed in an accident where automation was to blame, and so the uproar starts.

On a societal level, it is clearly better to have 1 person killed by automation than having three killed by human error.

I call BS. We're the ones who have to do the dying and grieving. Alpha Zero can beat me at chess, but it doesn't know it's playing, and wouldn't complain if I unplugged its power supply.

How do we know that an automaton-operated vehicle 'will halt' if directed to do so by a police officer?

What happens if a hacker replaces your car's code so as to lock you in your car, disable manual control, and take you on a death ride?

 

[Orodruin]

I am against a vehicle being operated on the public highway without its operator being at risk of his or her own human calamity just as I am when I'm in or on a vehicle on the public highway.

This is a complete non-argument solely based on your own feelings. It has absolutely nothing to do with actual safety or risk assessment but is purely emotional.

I call BS.

That's your prerogative, but you would be wrong.

How do we know that an automaton-operated vehicle 'will halt' if directed to do so by a police officer?

You can program it to. Regardless, this is completely irrelevant for the original question.

What happens if a hacker replaces your car's code so as to lock you in your car, disable manual control, and take you on a death ride?

This is also a non-argument not at all related to automation itself. If someone wants to kill you by tampering with your car, it is not going to matter if it is automated or not.

 

[anorlunda]

The public, honestly, is not fit to judge these things because it tends to judge based on anecdotal level evidence.

I entirely agree. But the question will become politicized. I am confident that it will be settled emotionally and irrationally. That's the way democracies work.

 

[Orodruin]

I entirely agree. But the question will become politicized. I am confident that it will be settled emotionally and irrationally. That's the way democracies work.

Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.

 

[Guineafowl]

Interesting comments on the ethical side, but I’m more interested in the pragmatics - How resilient are modern cars to sensor glitches, given that their systems now presume to operate the steering and brakes for you? These are the two most safety-critical mechanisms on the machine.

 

[jrmichler]

cars are increasingly going this way - lane departure, auto-brake, stop-start... We’re relying on electronic components placed in harsh environments (vibration, damp, heat cycles) to implement these features. A faulty lane departure could jerk you into oncoming traffic, auto brake could jam on and cause a pile-up, stop-start could wind you across a T-junction (if you’re in first gear and the sensor says you’re in neutral).

All of these features are building blocks toward autonomous vehicles. My understanding is that the strategic plan toward autonomous vehicles is fill the vehicle with sensors, run all of those sensors to a central computer, write software, then road test. When the road test finds edge cases not covered by the software, write additional code, and road test. Repeat until the results are "good enough".

I have problems with this approach. It guarantees that there will always be problems, some fatal. Troubleshooting will be difficult and expensive. After troubleshooting, the cost to repair will be high. Repair parts will go obsolete and become unavailable. And the software will be bloat code, with the problems that implies. Note that I am assuming that the hardware reliability problems mentioned by @Guineafowl will be solved before the hardware is released to production.

The Rand Corporation https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf calculated that over ten billion test miles is the minimum to prove that an autonomous vehicle is safer than a human driven vehicle. Toyota https://www.wardsauto.com/autonomous-vehicles/toyota-autonomous-cars-need-trillion-mile-reliability says that autonomous vehicles must be much safer than human driven vehicles, and that to prove this would require a trillion (1000 billion) test miles.

 

[Guineafowl]

When the road test finds edge cases not covered by the software, write additional code, and road test. Repeat until the results are "good enough".

I have problems with this approach. It guarantees that there will always be problems, some fatal.

I certainly agree with that. It means that the software updates for your car will quite likely be ‘written in blood’.

Attempting to test every scenario, and modifying software to suit, is what led to the 737 crashes. They missed out on the basics of engineering automated machines - redundancy and override.

Given the performance of today’s microcontrollers, especially those subject to harsh conditions (mechanical and electrical, eg high-voltage sparks, a ripply alternator output...) do we think there will be cars laid up by the side of the road while the drivers turn them off and on again?

And how about the effects of Bluetooth, mobile signal, etc on the car’s data lines? Sure, if the ABS computer gets disrupted the light comes on, but the brakes will still work when you press them. Once the car starts operating these controls for you, I think we need to start looking very carefully at comms resilience and fail-safe designs.

 

[anorlunda]

How resilient are modern cars to sensor glitches

What kind of answer are you looking for?
An adjective --- very/not much?
A number ---- 34827.4 * 10^-3?
An example --- ABS braking?

 

[Guineafowl]

What kind of answer are you looking for?
An adjective --- very/not much?
A number ---- 34827.4 * 10^-3?
An example --- ABS braking?

All three!

I was wondering if there’s anyone on here who’s worked on these systems and can say, for example:

“The auto-brake systems are very well designed. There are three sensors, and a majority vote is used. In the event they all disagree, the system shuts down and flashes a warning light. If the system otherwise malfunctions, there’s a big fat switch on the dash for the driver to turn off so he doesn’t have to bunny-hop all the way home.”

Is it unreasonable to expect this from a car? If the above had been applied to MCAS on the 737, things might have turned out better.

 

[anorlunda]

The thermostat senses temperature and regulates the flow of cooling water. If the temperature sensing function fails, so does the regulating function and the car may overheat or underheat. That has been the same since the first water cooled engines.

For the engines I owned, a thermostat fails about once in every 7.3 engine- years.

I would say that a thermostat is "somewhat" reliable.

There you go, all three types of answers.

If the system otherwise malfunctions, there’s a big fat switch on the dash for the driver to turn off so he doesn’t have to bunny-hop all the way home.
Is it unreasonable to expect this from a car?

How many big fat switches do you see on your dashboard today?

Forgive my sarcasm. Your thread sounds more like a rant than a question.

 

[Guineafowl]

How many big fat switches do you see on your dashboard today?

A metaphor - I meant an easily-disableable system.

Forgive my sarcasm. Your thread sounds more like a rant than a question.

Sorry, yes - I tend to write a little too much.

 

[LURCH]

Like most people, I have trouble deciding if computers should be allowed to operate motor vehicles. However, were I to base an evaluation strictly on statistical data and logic, I suppose I would easily conclude that humans should not. Given that cars are already out on the roads, and unlikely to go away any time soon, the only choice left is whether they are driven by people or machine systems. I consider the machines to be the safer of the two options available.

The potential hacker problem has occurred to me, as well. My greatest concern would not be about an individual trying to sabotage my vehicle, but a terrorist cyber attack on the networked system, turning thousands of vehicles into weapons simultaneously. However, I think block-chain technology would provide the cyber security necessary.

 

[essenmein]

The recent 737 tragedies have shown us, I think, a few pointers about automation of planes. Of course, investigation is ongoing but my understanding is that the input from a faulty angle-of-attack sensor and an automated system which the pilots couldn’t switch off contributed.

Two worries here - lack of multiple sensors to allow a ‘majority vote’, and no manual override.

Leaving aside driverless prototypes, cars are increasingly going this way - lane departure, auto-brake, stop-start... We’re relying on electronic components placed in harsh environments (vibration, damp, heat cycles) to implement these features. A faulty lane departure could jerk you into oncoming traffic, auto brake could jam on and cause a pile-up, stop-start could wind you across a T-junction (if you’re in first gear and the sensor says you’re in neutral).

1. Are we convinced there’s enough redundancy and resilience built into cars?
2. Do we need a simple ‘all-kill’ switch to allow total driver control, say in the event of a global CAN bus glitch?
3. Will there be a time when we need to put phones, etc into flight-safe mode before setting off?

1 and 2 are covered under ISO26262.

Which is to say the relevant function of the "thing" of interest is first analysed to understand its impact, then from there you determine the functional safety level needed, ASIL A/B/C/D or QM if it does not present a safety risk. ASIL D is the highest safety rating this would incl your electric steering system, ABS, or electric drive train (unwanted acceleration). ASIL A might be your rear tail lights if both fail, ASIL B is if your head lights or brake lights fail (both).

Then how you handle questions like redundancy is determined by the required ASIL level, D for example is typically dual lock step cores in the MCU, sometimes dual CAN, memory redundancy, if you have a sensor then somehow redundancy will be needed, which basically means putting two in running from separate supplies etc. It may include external shut downs, really depends on the system or safety goals.

 

[essenmein]

https://en.wikipedia.org/wiki/ISO_26262

 

[essenmein]

Like most people, I have trouble deciding if computers should be allowed to operate motor vehicles. However, were I to base an evaluation strictly on statistical data and logic, I suppose I would easily conclude that humans should not. Given that cars are already out on the roads, and unlikely to go away any time soon, the only choice left is whether they are driven by people or machine systems. I consider the machines to be the safer of the two options available.

The potential hacker problem has occurred to me, as well. My greatest concern would not be about an individual trying to sabotage my vehicle, but a terrorist cyber attack on the networked system, turning thousands of vehicles into weapons simultaneously. However, I think block-chain technology would provide the cyber security necessary.

Pretty much bang on, for me personally, the "should computers drive cars" is split in two, should they drive other peoples cars and should they drive my car. I think we all agree that all the other people should just learn how to drive properly. But it is personal, I like driving and I kinda don't want a world where I can't. I've seen the simulations, I'm not one of these I'll be the wolf in sheep's clothing person, I'm certain a human won't even be able to merge with properly automated traffic.

To me the only real argument against autonomous cars is cyber security, because there is no such thing, just levels of motivation. I also agree that its highly unlikely I'd be a target, but part of the 1k deaths due to the highway hack of 2027?

The low hanging fruit for autonomous driving where I think that it will happen sooner rather than later, that has decent economic incentive is long haul trucking and goods transport in general. The machines will cut their teeth driving on difficult terrain running farms and mines.

 

[JBA]

With regard to cities in the US that have autonomous vehicles are already being tested, and by whom, see the below PopSci article:

https://www.popsci.com/self-driving-cars-cities-usa

 

[FactChecker]

Has anyone considered a system that would monitor the driver, rather than drive the car? Suppose the system could take action if it detects a driver who is drunk, dozing off, distracted, or acting confused (driving the wrong way). IMO, that would be much more beneficial and easier to implement.

 

[anorlunda]

Has anyone considered a system that would monitor the driver, rather than drive the car? Suppose the system could take action if it detects a driver who is drunk, dozing off, distracted, or acting confused (driving the wrong way). IMO, that would be much more beneficial and easier to implement.

I think some of that already exists in production. But I apologize, I have no links to back that up right now.

 

[Borg]

Has anyone considered a system that would monitor the driver, rather than drive the car? Suppose the system could take action if it detects a driver who is drunk, dozing off, distracted, or acting confused (driving the wrong way). IMO, that would be much more beneficial and easier to implement.

Didn't the 737 Max have something like that?

But seriously, there are plenty of cars out there right now with various features such as lane departure warnings, auto-braking, etc. AFAIK, there isn't much that specifically observes the driver but there are still plenty of other ways that automation can improve safety.

 

[Nik_2213]

Disclaimer: Personal Experience.
My wife's very nice new car had many real-neat safety features, including front and back 'park assist' sensors that doubled as tail-gating alerts and crash mitigation.

Trivial problem: parking in supermarket bay usually woke 4~5 of the six sensors, so sounded like a raucous flock of chickens...

Existential problem: I was driving along when a gust of wind swayed a road-side bush over edge of lane. Car automatically applied the anti-lock brakes, full on. How following traffic missed us, I do not know...

My initial thought was the car had suffered a blow-out, suspension collapse or CV drive seizure. My wary walk-around found nothing amiss. I was still scratching my head, wondering if it was safe to drive on, when another gust of wind swayed that bush again, and I understood...

FWIW, the dealership salesman had mentioned being stranded one evening when his similarly equipped city-car halted without warning, refused to go forwards. There was no obvious cause. The 'Rescue' crew checked the car up, down and sideways, then peeled a transparent candy wrapper from a front sensor...

Yes, I could delve the menus and disable this facility, but insurance provider made it clear that active sensors would significantly mitigate their liability, so please leave them on...

 

[anorlunda]

Disclaimer: Personal Experience.

Thanks for the reminders. Hopefully, the struggle between complexity and the KISS principle will never end.

 

[256bits]

Disclaimer: Personal Experience.

Fix one problem.
Create more problems.
I think your experience(s) says it all.