Problems with the Dreamliner battery

[berkeman]

I quoted someone, to that effect, the other day.

I cannot find my quote, nor who else echoed that...

I looked back a bit in your posting history and couldn't find it yet. Do you know which day you did it on?

 

[berkeman]

Ah ha!

Just did the experiment.

Do not try this at home. Although I didn't die, you will...

In trying to understand how such a thing could happen, I tried to figure out the resistance of seawater, as the google answers to my question, used bizarre, unheard of terms:



Anyways, I mixed together a solution of seawater(1 pint of water and 1 tblsp of salt in a pyrex measuring cup), and tried to measure the resistance with my VOM: 1.2kΩ on the 200kΩ scale, and 85kΩ on the 2MΩ scale. Complete nonsense.

So I hooked up my marine battery in series with a 1Ω resistor with my 1 pint of seawater and came up with a value of ≈11Ω.

I decided that the 11Ω value was irrelevant to the problem, as while I watched, Hydrogen and Oxygen gasses were being emitted at a non-insignificant rate. I didn't dilly-dally in deciding that this experiment could have very bad results, as my 6 gallon gasoline tank was tethered to my battery, and less than a foot away. A spark would have resulted in the HHO mixture igniting, potentially igniting any fumes from the gasoline tank.

I disconnected the experiment, in a thoughtful manner, as, an acquaintance of mine, while trying to retrieve his wallet from the dashboard of his burning car, was unfortunate enough to do that, the moment his car burst into flames.

He never looked the same after that. He looked a little better after the plastic surgeons made him a new nose, but not much.

We're going to need to change your username to Danger. Oh wait...

 

[the_emi_guy]

Ah ha!

Just did the experiment.

Do not try this at home. Although I didn't die, you will...

In trying to understand how such a thing could happen, I tried to figure out the resistance of seawater, as the google answers to my question, used bizarre, unheard of terms:



Anyways, I mixed together a solution of seawater(1 pint of water and 1 tblsp of salt in a pyrex measuring cup), and tried to measure the resistance with my VOM: 1.2kΩ on the 200kΩ scale, and 85kΩ on the 2MΩ scale. Complete nonsense.

So I hooked up my marine battery in series with a 1Ω resistor with my 1 pint of seawater and came up with a value of ≈11Ω.

I decided that the 11Ω value was irrelevant to the problem, as while I watched, Hydrogen and Oxygen gasses were being emitted at a non-insignificant rate. I didn't dilly-dally in deciding that this experiment could have very bad results, as my 6 gallon gasoline tank was tethered to my battery, and less than a foot away. A spark would have resulted in the HHO mixture igniting, potentially igniting any fumes from the gasoline tank.

I disconnected the experiment, in a thoughtful manner, as, an acquaintance of mine, while trying to retrieve his wallet from the dashboard of his burning car, was unfortunate enough to do that, the moment his car burst into flames.

He never looked the same after that. He looked a little better after the plastic surgeons made him a new nose, but not much.

Glad you made it back safely!

You are right, 12V + seawater will not burn due directly to the electrical load of the seawater.
What happens is that corrosion (take a look at your contacts that were in the water) can cause low resistance shorts which then can then burn. Their creepage distance requirements probably did not anticipate being submerged.

 

[OmCheeto]

Glad you made it back safely!

You are right, 12V + seawater will not burn due directly to the electrical load of the seawater.
What happens is that corrosion (take a look at your contacts that were in the water) can cause low resistance shorts which then can then burn. Their creepage distance requirements probably did not anticipate being submerged.

If it had been any other car, I probably would not have researched this incident. But I knew Fisker used A123's batteries, and they were one of the safest lithium batteries on the market.

GM turns to A123 Systems for batteries less volatile than Chevrolet Volt’s
December 08, 2011
...
GM is using phosphate-based lithium-ion batteries from Waltham, Massachusetts-based A123 Systems Inc. that are less likely to burn than other lithium chemistry, including that used in the Volt model that went on sale last year, according to the companies. ...

It should be noted that I still haven't sold my A123Q, aka B456, stocks. I foolishly assumed that trading of the stocks would cease once the company went bankrupt. Unfortunately, I was so disgusted with the whole incident, I paid no attention to my portfolio, which was set to auto-purchase shares each month, and only discovered later that I had doubled the number of shares I owned in just a few months. If I hadn't shut that down, it would have doubled again by now.

Oh dear... I see I now own another Q suffixed stock! XIDEQ...

I guess the battery market is as brutal as the solar panel market.

ps. The battery that I did my seawater test with is an Exide.

 

[AlephZero]

Another dreamliner electrical screw-up. Engine fire extinguishers wired the wrong way round on three aircraft in service.

http://www.bloomberg.com/news/2013-...-in-dreamliners-as-jal-plane-scraps-trip.html

That wouldn't be funny if there was a problem that caused an engine fire. The engine fire extinguishers will put out most things, but they are pretty much a one-off last resort. Discharge a fire bottle into the one engine still working, and you had better be qualified as a glider pilot.

 

[jim hardy]

Oh dear... I see I now own another Q suffixed stock! XIDEQ...

don't feel bad I still have Enron.

 

[Ptero]

http://newscenter.lbl.gov/wp-content/uploads/Balsara-dendrite-growth.jpg
Here we go again. Dendrites grow slowly over time like fractures.

Researchers with the U.S. Department of Energy (DOE)’s Lawrence Berkeley National Laboratory (Berkeley Lab) have discovered that during the early stages of development, the bulk of dendrite material lies below the surface of the lithium electrode, underneath the electrode/electrolyte interface. Using X-ray microtomography at Berkeley Lab’s Advanced Light Source (ALS), a team led by Nitash Balsara, a faculty scientist with Berkeley Lab’s Materials Sciences Division, observed the seeds of dendrites forming in lithium anodes and growing out into a polymer electrolyte during cycling. It was not until the advanced stages of development that the bulk of dendrite material was in the electrolyte.
http://newscenter.lbl.gov/science-shorts/2013/12/17/roots-of-the-lithium-battery/

Dreamliner Grounded As White Smoke Spotted -- TODAY!
http://news.sky.com/story/1195209/dreamliner-grounded-as-white-smoke-spotted

 

[nsaspook]

Another Telsa crash and fire but this one has a 'twist'.

http://ktla.com/2014/07/04/tesla-cr...lits-in-half-in-weho-multiple-people-injured/

You can see flaming parts of the broken battery pack ejecting flaming cells like missiles but the car looks to be incredibly safe if you are free of the resulting fire.

The first part gives a good view of the scale of the wreck. It looks like the driver was trying to turn or slid at high speed after hitting the Honda and hit the pole near the middle of the car spilling most of the battery in the street. Losing that heavy battery might have saved his life.
https://www.youtube.com/watch?v=kE_u731EmYA

 

[OmCheeto]

...
Losing that heavy battery might have saved his life.
...

Not driving any stolen car at 100 mph down the streets of LA, and not self T-boning a light pole, will probably also save your life. (I read that he actually took out two light poles.)

Latest reports say the the thief is still alive, though one of the people he injured is still in critical condition.

I consider this a non-issue.

 

[AlephZero]

If the battery falls out of the car in a side-on collision, I think that's a major design issue in its own right, independent of any pyrotechnic side effects.

 

[OmCheeto]

If the battery falls out of the car in a side-on collision, I think that's a major design issue in its own right, independent of any pyrotechnic side effects.

The car hit the pole with enough force to slice the car in two. Looking at the layout of the battery, and knowing there are over 7000 individual cells that make up the battery, I'm guessing the "battery" did not fall out of the car.

The gas tank on my truck is about 5 feet long and is positioned centrally, running parallel to the drive shaft. Would that also be considered a major design issue if 17 gallons of gasoline were released in a similar accident? Both the Tesla and my Truck have 5 star crash ratings.

On the following website, they claim:

Pole Side Impact
...
In the test, the car tested is propelled sideways at 29kph (18mph) into a rigid pole.
...

I can go faster than that on a bicycle.

How much would it add to the cost of vehicles if the speed of the test was increased to 100kph (62mph)? Do we really need to engineer vehicles to keep people alive, based on the driving habits of contenders for the Darwin Award?

Consensus at the Oregon Electric Vehicle Assn about this incident? : Yawn

 

[AlephZero]

The car hit the pole with enough force to slice the car in two.

OK, if you think that was the root cause, then that's still a major design issue.

Crash test regulations may or may not be a marketing exercise, but if the car hit a standard item of highway engineering (I'm not familiar with what the US term "pole" signifies here) while being propelled by its own engine, that event is going to happen sooner or later.

I'll be interested to see the first head-on crash between two Teslas with a closing speed of around 200 mph...

 

[nsaspook]

Consensus at the Oregon Electric Vehicle Assn about this incident? : Yawn

They're mainly right. A ICE car could have turned into spinning firebomb also in those conditions. My main point with the post was to demonstrate the reaction of some types of lithium batteries is very similar to carbon based fuels in these extreme conditions with the added punch of not being able to use normal extinguishers like water or water based products to control the fires. The firefighters just letting it burnout was the safest approach. With an isolate car in the street that's a good option if no one is trapped inside, in a flying plane filled with people it's not.

Don't do this
https://www.youtube.com/watch?v=ojGaAGDVsCc

(We have class D bottles at work for hydride metal fires, nasty stuff but we don't have to use the Lithium approved types)
http://www.safetyemporium.com/ILPI_Site/WebPagesUS/detail.htm&&2eiBpe0obDuox2NvxMpoLGxolobo2bbBeJ1gctPw1JaImsaa2r1GcDQUR4Vh_S6alkWaTWuala?09584

 

[AlephZero]

How much would it add to the cost of vehicles if the speed of the test was increased to 100kph (62mph)? Do we really need to engineer vehicles to keep people alive, based on the driving habits of contenders for the Darwin Award?

First, decide what crash tests are supposed to be testing for.

For example, the US crash tests for fenders/bumpers are impacts with a rigid barrier at 3mph and 6mph, designed to measure the damage to the car.

The corresponding EU test is impact with a deformable object at 25mph, designed to measure the damage to the object - e.g. a pedestrian.

 

[OmCheeto]

... (I'm not familiar with what the US term "pole" signifies here) ...

Odd, as I quoted a european website, regarding the "pole" test.

About us
Established in 1997, Euro NCAP is composed of seven European Governments as well as motoring and consumer organisations in every European country. (Read more on our members)

I'll be interested to see the first head-on crash between two Teslas with a closing speed of around 200 mph...

That would be fun to watch. Darwinians are quite entertaining, regardless of their energy source.

 

[Greg Bernhardt]

Dreamliner battery probe ends: 8 questions and answers
http://www.cnn.com/2014/12/11/travel/boeing-787-dreamliner-investigation-report/index.html?hpt=hp_t2

 

[Doug Huffman]

NTSB report from the CNN article linked above
http://i2.cdn.turner.com/cnn/2014/images/12/10/air1401.pdf 2MB 110 pages

 

[Borg]

Just when you thought it was over... To keep a Boeing Dreamliner flying, reboot once every 248 days. What the heck did they do - give it a Windows operating system?

According to the FAA, there's a software bug in the 787 Dreamliner that can cause its electrical system to fail and, as a result, lead to "loss of control" of the plane. But why? The FAA says this is triggered by the aircraft's electrical generators, which could give out if they have been powered on continuously for over eight months.

 

[rollingstein]

Just when you thought it was over... To keep a Boeing Dreamliner flying, reboot once every 248 days. What the heck did they do - give it a Windows operating system?

Stole this from a comment elsewhere but sounds most plausible:

"248 days == 2^31 100ths of a second. even in 2015, our airplanes have integer overflow bugs "

 

[russ_watters]

One of my solid state drives had nearly exactly the same problem. After a certain number of run hours -- something like 248 days -- it turned-into a a brick. Fortunately, it had a reset routine and bios flash to fix it.

Now with the dreamliner it says "continuously powered", so I suppose you could just reboot it once every 8 months to work-around this bug.

 

[nsaspook]

One of my solid state drives had nearly exactly the same problem. After a certain number of run hours -- something like 248 days -- it turned-into a a brick. Fortunately, it had a reset routine and bios flash to fix it.

Now with the dreamliner it says "continuously powered", so I suppose you could just reboot it once every 8 months to work-around this bug.

Have none of the programmers heard of signed integer overflow bugs? At 100hz 248.5 days would be just about right. One woodpecker!

 

[anorlunda]

Re: Newest Dreamliner Bug.

Once again, this proves that all the kings horses and all the kings men can never make flawless yet complex software. There is no such thing and there never will be.

I'm afraid that the airline industry will be doomed to follow the nuclear industry. During Y2K remediation reviews, we found software written in the early 1960s still in use, and still being replicated from the oldest plants into the newest ones. The source code, who wrote it, what its function was, and how it works are all unknown, but nobody dared to touch it. You see, when software is in operation for 50 years without failure it attains the status of "proven". No matter how many latent bugs remain in the proven software, we are pretty sure that they will never be triggered or cause a problem. Anything new, regarless of the time and money spent to perfect it, is "unproven."

Therefore, I predict, that after 10 or so years pass without new Dreamliner bugs causing failures, that nobody will ever dare touch it thereafter. Even if a previously unknown bug causes a crash 25 years from now, the risk of any remediation will be greater than the risk of doing nothing.

 

[rollingstein]

Once again, this proves that all the kings horses and all the kings men can never make flawless yet complex software. There is no such thing and there never will be.

Agreed.

Therefore, I predict, that after 10 or so years pass without new Dreamliner bugs causing failures, that nobody will ever dare touch it thereafter. Even if a previously unknown bug causes a crash 25 years from now, the risk of any remediation will be greater than the risk of doing nothing.

Not sure about never touching it. Look at the Linux kernel. It could very likely be one of the most complex software codes out there. But bugs are still getting regularly fixed. And Linux kernels get eventually used in many fairly critical tasks.

So I guess although we can never certify complex code as "bug free" that doesn't stop us from carefully fixing bugs.

 

[jim hardy]

Diminishing returns is a natural law .

On the other hand, it remains to be seen what will be the effect of this structure imposed by the machines on human language and thought.

I was the last of the "Slide Rule Generation" and well remember when Miami had only one TV station that was on only afternoon and evenings.
I've seen the "Sesame Street Generation" come to power in society.

Check out code.org and try their "hour of code" exercise - it's made for little kids but it's an absolute blast.
I wish i could live long enough to see what these computer generation kids do.

old jim

 

[jim hardy]

oops- that was a hijack, wasn't it ?

They should fix that integer overflow.
and make sure it's a global variable.

 

[anorlunda]

Not sure about never touching it. Look at the Linux kernel. It could very likely be one of the most complex software codes out there. But bugs are still getting regularly fixed. And Linux kernels get eventually used in many fairly critical tasks.

That's a different scenario. If Linux was used for a critical task, and if it had not been touched for 10 years, and if nobody today had experience changing Linux in the past 10 years, would you change it then?

 

[rollingstein]

That's a different scenario. If Linux was used for a critical task, and if it had not been touched for 10 years, and if nobody today had experience changing Linux in the past 10 years, would you change it then?

The Linux kernel gets already used for critical tasks. Well, depends on what your definition of critical is. Not where Real Time OSs are needed but Critical tasks nevertheless.

The "not touched for 10 years" point I agree with you. But will codes like the aircraft software go untouched & unmaintained for 10 years? Is the source really as ill-documented as you mention for nuclear industry codes?

Software engineering in the 1960's was a totally different beast than when the Dreamliner codes got written.

 

[anorlunda]

The Linux kernel gets already used for critical tasks. Well, depends on what your definition of critical is. Not where Real Time OSs are needed but Critical tasks nevertheless.

The "not touched for 10 years" point I agree with you. But will codes like the aircraft software go untouched & unmaintained for 10 years? Is the source really as ill-documented as you mention for nuclear industry codes?

Software engineering in the 1960's was a totally different beast than when the Dreamliner codes got written.

Good points. Time will tell.

But eventually, someone will notice that most Dreamliner bugs found in service result from the most recent software changes, and that someone will blow the whistle. Think of incidents such as the recent nationwide outage at Starbucks. Consider the risk-benefit ratio if auto manufacturers broadcast updates to all those embedded microcontrollers in cars. When things run well, there are strong incentives to leave them alone.

I would be interested to hear how many embedded Linux apps run with the software initially installed (perhaps burned in ROM) and never updated. I have a GPS chartplotter on my vessel that fits that description. It's function is pretty critical.

 

[rollingstein]

But eventually, someone will notice that most Dreamliner bugs found in service result from the most recent software changes, and that someone will blow the whistle. Think of incidents such as the recent nationwide outage at Starbucks. Consider the risk-benefit ratio if auto manufacturers broadcast updates to all those embedded microcontrollers in cars. When things run well, there are strong incentives to leave them alone.

I would be interested to hear how many embedded Linux apps run with the software initially installed (perhaps burned in ROM) and never updated. I have a GPS chartplotter on my vessel that fits that description. It's function is pretty critical.

I think purely aesthetic patches will be frowned upon. Or patches that add marginal functionality. But what about known bugs & the resultant bug fixes? Often a bug fix can trigger an unintended, unanticipated disaster.

What's the empirical observation, though? Are most inconvenient / massive bug-related-down-times the result of proximal updates?

Given the combinatorically large number of states a complex software can be, isn't it about as likely that a bug latent since inception gets accidentally triggered at some later time just by pure chance leading to disastrous consequences? Without being the result of any new updates really.

Ergo, is ROMing critical software & never updating it a better option in practice?

 

[donpacino]

Stole this from a comment elsewhere but sounds most plausible:

"248 days == 2^31 100ths of a second. even in 2015, our airplanes have integer overflow bugs "

A few things to consider about aircraft control computers.
There are independently reset-able paths usually using dissimilar hardware. Resetting a CPU or event an entire board will barely effect system operation.

Most electrical systems on a modern airplane reset once or twice a flight due to single event upsets (SEUs).

 

[rollingstein]

A few things to consider about aircraft control computers.
There are independently reset-able paths usually using dissimilar hardware. Resetting a CPU or event an entire board will barely effect system operation.

Most electrical systems on a modern airplane reset once or twice a flight due to single event upsets (SEUs).

Very interesting. In this context when you talk about an electric system resetting itself what is the smallest resettable unit? e.g. when a SEU happens how large is the system that gets reset?

Or when you talk about resetting a Dreamliner CPU, how many CPUs are on board in the first place?

 

[donpacino]

Very interesting. In this context when you talk about an electric system resetting itself what is the smallest resettable unit? e.g. when a SEU happens how large is the system that gets reset?

Or when you talk about resetting a Dreamliner CPU, how many CPUs are on board in the first place?

So a single flight computer system is typically made up of multiple boards. You can typically reset on a box (multiple boards), board(multiple functions), or function level. examples of functions are com channels, watchdog channels, and flight mode channels. You can sometimes even reset individual memory blocks (which will most likely disable that entire function for that frame). I would say the smallest electronic system you can reset is a function.

I don't know the dreamliner's architecture, but think of it this way. If there are 3 redundant paths in each computer, and you have say 8 actuators computers, 2 mission/flight computer, at a minimum you'll have 30 flight critical cpus. I would say that's a LOW estimate.

 

[rollingstein]

You can typically reset on a box (multiple boards), board(multiple functions), or function level.

box & board resets I can understand but what does a function level reset mean? Or even a memory reset. i.e. If you reset memory how do you get a coherent state for the OS wherever it was last before the reset? Ditto for a function reset. A function is a software entity right? So this would be a soft reset?

 

[donpacino]

box & board resets I can understand but what does a function level reset mean? Or even a memory reset. i.e. If you reset memory how do you get a coherent state for the OS wherever it was last before the reset? Ditto for a function reset. A function is a software entity right? So this would be a soft reset?

I shouldn't have used the word function. By function I meant a 'block' or a circuit that preforms a particular task. An example of this is on a typical PC if you had the ability to reset the hardware (not the software) of an individual usb port.

with the memory reset issue you may need to restart the OS or some higher level function depending on what part of memory was affected. In flight critical applications all or most memory is partitioned for specific use. Even then, the reset times for most systems are not that long (hard resets are anywhere from 10 ms to 200 ms). software can be much longer (seconds). But compared to windows operating systems, its blazing fast

 

[rollingstein]

Are the single event upsets so much more often due to the altitude of flights & related increase in energetic particle bombardment?

 

[donpacino]

Are the single event upsets so much more often due to the altitude of flights & related increase in energetic particle bombardment?

I don't know that much about SEUs. I know what they are. I know how they effect electronics. I know that the thinner the atmosphere, the more prevalent they are.

 

[rollingstein]

I don't know that much about SEUs. I know what they are. I know how they effect electronics. I know that the thinner the atmosphere, the more prevalent they are.

How come they don't cause laptops or similar electronic devices that have no special protection against SEUs to keep crashing when used on aircraft?

 

[nsaspook]

How come they don't cause laptops or similar electronic devices that have no special protection against SEUs to keep crashing when used on aircraft?

The sensors, wiring and interfaces are exposed to high energy transients from power control activators and hull discharges from static. A person sitting at a seat is usually well shielded by the cabin and are far away from most EMI/ESD sources.