Industry Embedded Software Development Environments

[.Scott]

TL;DR

In the wake of a dead battery, a car problem lead to this discussion related to Embedded Software Development.

This post was made in another thread in the "General Engineering" forum in a thread related to automobiles:

Software. A reliable RTOS is a myth ASFAIK. The exception is if it cost as much as what they put in an F-18 and such. We simply don't pay enough to get really good SW in our cars.

An automobile "Electronic Control Unit" (ECU) is an embedded computer processor module responsible for all sorts of data management chores in modern cars. Each manufacturer is responsible for selecting and/or designing the ECU for each of their car models. These units run highly customized software and may or may not use an "off-the-shelf" "Real-Time Operating System" (RTOS).

In the US (and many other countries), the ECU software must be developed under "MISRA" standards.

Personally, I have never programmed an ECU, but I have done a lot of Software development work on automotive radar units.
For that unit, the RTOS was kind of a starter kit. The developers would supply scores and scores of pages of parameters, and the package would generate source code for an RTOS finely tuned to what we would be needing. As a member of the "core" group, if we found a problem in the generated code, we would fix it. There were about eight of us working on this for roughly 5 years. For the most part, issues with the RTOS were resolved within the first 12 months. After that, there were rare changes in response to things like the electronic interface to the rest of the car.

These radar units sold in the tens of millions of units - so "Non-recurring Engineer" (NRE) costs were not a major obstacle.

I don't know if there is any one ECU that is used in as many cars as this radar (Veoneer).

In general, embedded software has the potential for being very reliable. In many cases, the OS is simply too small to hide major surprises. And in most cases, the support for the application has a very limited and predictable scope that allows for thorough testing.

So, if your car needs to "recover" from a dead electrical system, it was likely a deliberate management/engineering decision.

 

[berkeman]

So, if your car needs to "recover" from a dead electrical system, it was likely a deliberate management/engineering decision.

I didn't read much of the other thread, so maybe this has already been mentioned -- There are ways that the data that is stored in such embedded systems can get corrupted, and that can lead to confused operation (or even bricking). If the device is designed well, it can recover from the bad data as more new data is acquired (which is what it sounded like happened in that car thread when the problem went away in a couple of days).

In order for a dying battery to not cause corruption of important data (timing information, fuel mixture information, driver habits information, etc.), you need at least a good low-battery-sensing circuit for early warning of the failure, and you need a good flash file system to manage the non-volatile memory that holds the non-volatile data. Getting all that to work is non-trivial in my experience.

 

[DaveE]

So, if your car needs to "recover" from a dead electrical system, it was likely a deliberate management/engineering decision.

Or ignorance and wishful thinking. Many in the C-suite just don't appreciate how hard it is to make big SW projects work well. Others have been mislead by the online world of Silicon Valley; move fast and break things only really works well if you can send out SW updates whenever you find that you haven't really finished the job. Outsourcing SW testing to your installed customer base doesn't work for HW. Testing large SW is really hard and often not given sufficient time. Time to market is the Achilles heel of big embedded systems.

 

[.Scott]

As I said I never programmed an ECU. But I did have a Chevy Conversion Van that ran into this problem. It minimally start up to an idle, then you had to carefully apply gas and load to get it to move for a minute or two. In that amount of time, it had relearned/recalibrated and I was literally good-to-go. This was about 15 years ago.

It's pretty trivial to detect bad data. Maintaining a 64-bit CRC checksum will do the trick. If on ignition, it's bad, go into a safe training mode.

Time-to-market is definitely an issue with many embedded systems. But not so much with consumer trucks and automobiles. At the same time you have a team on the software, you have other teams putting the supply chain and assembly line together.

On the other hand, MISRA rules are often implemented in a most onerous way.

And what might appear to be an elegant solution may fail to safely address all failure modes.

 

[Averagesupernova]

None of the data recovery tricks matter when there is a hardware issue that causes the ECU to retune the engine to run fairly close to correctly. At this point the check engine light is on. When the ECU is hooked to the scan tool and reads out the problem, the hardware is repaired and and now the engine runs horribly until it relearns and retunes. Probably most noticeable on diesel engines.