Grim Day on the Texas Power Grid

[anorlunda]

You make several good points.

1) A smaller grid with too small reserve capacity is more likely to fail than a larger one with more backups, this I think is pretty much a universal axiom

That is undeniably true, and it is why the interconnections in the rest of North America are so big. But in this case, Texas' neighboring states had similar problems on the same day (see post #3 in this thread). That makes it difficult to say that the problem was entirely due to causes unique to Texas, or whether the neighbors could have helped much.

2) whenever a substantial portion of the grid is made by renewables like solar and wind one also has to have failsafe backups that can act as base loads much like a nuke plant has its diesel's always ready.

That is also true. But in this case, renewable wind was scheduled to provide only 10% of the generation on that day, while the shortfall was 40%. So although a contributor, renewables weren't the biggest contributors.

Actually natural gas I think is also a good reserve as well as active , not sure how much of it Texas has and in what condition

Texas has huge quantities of natural gas. Texas supplies most of North America with natural gas. But the gas pipelines themselves froze, causing gas supply to fail to many of the 158 power plants that went down and to many private homes that have natural gas heat and/or natural gas fueled backup electric generators. So, what happens when your upstream suppliers also fail? Natural gas is like electric energy. You can't store 3 month's supply in advance.

I can draw some similarities to Fukushima,

Now they say the Texas power grid is made from Swiss Cheese, and all the holes lined up.

I too thought of Fukushima. The article about the Swiss Cheese model of reliability is just another name for common mode failures. Common mode failures are the bane of reliability engineers, because the rules for calculating the probability of multiple independent events don't apply if they are not independent. In Fukushima, even though they had multiple redundant diesel generators, they had a common diesel fuel storage that got contaminated with salt water.

If a common mode failure causes highly engineered things to fail, the only thing we can say is that the imaginations of the engineers was too limited. How can you account for something that you have not imagined?

Whenever I see the private house construction videos from US south side I see thin structures that are not made for any real thermal insulation for longer than a few hours.

"Rolling blackouts" are supposed to be designed so that no customer is out for more than 3-4 hours (they said 12 hours in the WSJ article). Even in thin walled houses, it may take longer than that for pipes to freeze. But this week, some customers were out for more than 72 hours. That indicates a second level failure. The rolling blackouts didn't roll.

when everyone switches on their heater the grid drops.

Most modern houses in the area have central heat pumps to provide air conditioning cooling and heating. When the heat pumps can't keep up, they have resistive heating elements that come on automatically to provide supplementary heat. Those switch on and off to control temperature. If the supplementary need is slight, only a few of them will be on at the same time. But if it is so cold that more supplementary is needed than available, all of them will stay on all the time. I'm sure that was what caused a record high demand for that day. A record high demand simultaneous with a 40% shortfall in supply seems to be a perfect storm of improbable events. But if they have a common cause it is not so improbable at all.

A giant asteroid crashing into Earth could cause a world-wide blackout. But we wouldn't care, because we would all be dead anyhow. Sorry for the black humor, but it illustrates a point. Wild flights of imagination about common mode events can also lead to absurdity. Overactive imaginations IMO cause much of the anti-nuclear feelings around the world. We can both underestimate and overestimate risks with bad consequences in both extremes.

 

[austinuni]

Whenever I see the private house construction videos from US south side I see thin structures that are not made for any real thermal insulation for longer than a few hours.

Good insulation is needed year round here in Austin (and in large parts of Texas) with our 100F+ temperatures during the summer, and frequent 40F temperatures during the winter. Texas is definitely not Florida or Southern California.

 

[256bits]

Power grids are filled with holes that all line up on a regular basis.

Agreed.

While the relay failure broke power for customers, that was easily fixed.
There was still a whole slew of other problems to deal with across the province . The ice was 3 - 4 inches thick, with some places much much more, downed HV power lines, and bent trees onto the lines.

So even if the Texas system is 'designed for the hot summer months', and failed in the winter, the Quebec system, 'designed for cold winter months', also failed in the winter.
Several areas, without power for months, mainly the Eastern Townships, would be akin to what the Texans are going through.

https://www.inmr.com/looking-back-on-the-great-ice-storm-of-1998/

Images example
https://www.google.com/search?q=quebec+ice+storm+images&rlz=1C1KMZB_enCA593CA862&tbm=isch&source=iu&ictx=1&fir=uHhpqtGfuiW7aM%2CCPY5pdVrOC7dFM%2C_&vet=1&usg=AI4_-kT0E0DVsMK1NGmTopYeJsFuAjVcaQ&sa=X&ved=2ahUKEwiu3NbxsPnuAhWgGFkFHZ7_DsEQ9QF6BAgFEAE#imgrc=uHhpqtGfuiW7aM

insulation - in two days time, our unit dropped in temperature to 40 degrees or so - we were sleeping in our winter parkas. One would have to foresee a lengthy power failure and "overinsulate" for such events. Few homes are made that way wherever they are. One builds for the normal climate and possible risks ( earthquakes perhaps) where one lives.

 

[anorlunda]

That same ice storm (it was 1998??) also hit northern NY, VT, NH, ME. I remember hearing the number 400,000 power poles snapped.

A huge victory for the grid in that storm is that the blackout did not cascade into areas not covered in ice. The rest of the grid survived roughly 1 million simultaneous failures. I remember citing that number a year later in 1999, when the Y2K fear mongers insisted that multiple simultaneous failures would spell doom for the power grid. It still circulates today when fear mongers cite cyberattacks as the thing that will plunge everyone into darkness for a year or more.

 

[DaveE]

I won't accept any simple explanations until I read more details from the investigation. It is analogous to a plane crash. There can be a chain of events and multiple contributing causes.

Yes, the devil is in the details. OTOH, at the most basic level, it isn't so complex. It's a question of how much planning and money is spent to build a resilient system. This isn't an accusation or judgement of those choices. The only bad choice is none. There are always pros and cons. But, people do have to live with the consequences of those choices regardless.

I think the interesting part of this is the fall out regarding public attitudes towards regulation vs free market; cheap energy vs. reliable supply; collective solutions vs. isolation from other states, etc. Texas has been an outlier in these choices. Will that continue?

 

[anorlunda]

Yes, the devil is in the details.

Read #29.

 

[nsaspook]

I think the interesting part of this is the fall out regarding public attitudes towards regulation vs free market; cheap energy vs. reliable supply; collective solutions vs. isolation from other states, etc. Texas has been an outlier in these choices. Will that continue?

Whatever solution is found it's likely to be uniquely Texan.

https://www.marvel.com/characters/texas-twister
They're like the Hulk, the more you pound them, the meaner they get.

 

[Tom.G]

It seems that Texans (or at least their politicians) value independence over reliability.

Here is an excerpt from an article in the Texas Tribune after the 2011 failure:
https://www.texastribune.org/2011/02/08/texplainer-why-does-texas-have-its-own-power-grid/

"The Texas Interconnected System (http://tx.findacase.com/research/wf..../FDCT/NTX/1979/19790130_0000007.NTX.htm/qx)— which for a long time was actually operated by two discrete entities (http://www.ercot.com/about/profile/history/), one for northern Texas and one for southern Texas — had another priority: staying out of the reach of federal regulators. In 1935, President Franklin D. Roosevelt signed the Federal Power Act, which charged the Federal Power Commission with overseeing interstate electricity sales. By not crossing state lines, Texas utilities avoided being subjected to federal rules. "Freedom from federal regulation was a cherished goal — more so because Texas had no regulation until the 1970s,""

A case of You Reap What You Sow.

 

[artis]

Well speaking about freedom sometimes people I think get rather foolish about it. I mean having freedom in a sense that you can have free speech or choice of religion is good but connecting a bunch of wires and circuit breakers won't suck anyone's soul out or make anyone possessed so I really can't see any downside of that , after all it's the same country that you are connecting to.I don't claim to be an expert and even though I drive a LPG car I haven't read that much about gas physics although I think I know some basics.
One thing seems weird to me, even though LPG is much better at being stored cold than natural gas aka methane, I think a gas all by itself cannot freeze , well not in mild cold conditions, it could liquefy but that would require temperatures nearing superconductor working ones.
Could it be that the Texas pipelines were not up to task and "clean enough" and that caused the freezing?
One way I can imagine this could occur if the gas is contaminated with water vapor which could then freeze up causing the pressure to drop in the pipe causing further cooling and freezing.
Here is a short paper describing something like that.
https://asgmt.com/wp-content/uploads/pdf-docs/2011/1/T06.pdf
Like a common failure mode in cars equipped with LPG is that the gas reductor (for those that don't know, the part were the liquefied high pressure gas is allowed to expand so that it can be turned back to gas state for dosing into individual cylinders. Because squirting in drops of liquid gas on hot metal piston would likely cause the engine to explode itself) so if this reductor loses it's heating the gas causes it to freeze to very low temperatures even during hot summer temperatures. Because the gas is performing the same process it does in a AC or any other cooling pump taking away heat as it expands.
So what happens is the water vapor surrounding the reductor freezes up and causes the gas pipes to become one giant frozen water clump.
This actually happened to me once. Then I went and repaired and rerouted my heating hoses. You could say I had a Texas happening under my hood...
Also as this happened the gas that entered my engine became more and more liquid and I could hear how detonation was starting to happen.

 

[artis]

I think @austinuni made a good point because one wants good thermal insulation year round because it not only helps keep the heat in at winter but the also keep the heat out in summer. Otherwise you are just making the local energy company rich by needlessly driving your AC.

 

[Rive]

1) A smaller grid with too small reserve capacity is more likely to fail than a larger one with more backups, this I think is pretty much a universal axiom

That's a tricky one. While in itself it's true: in practice it's often used to reduce the amount of reserves (especially in case some heavily donated intermittent sources pressuring the grid), so overall you can easily end with just bigger blackouts. As it went in Europe, a continent wide blackout is no longer just a nightmare - it's already an existing danger.

 

[anorlunda]

A case of You Reap What You Sow.

Let's not ignore the evidence. Post #3 in this thread shows that blackouts also occurred the same day in Oklahoma, Texas' nearest neighbor. (And other states too. See #3). Oklahoma is part of the large Eastern Interconnect in North America. Oklahoma also has it's own laws/regulations/operator/owners/profit motives/ ... and so on.

Based on that evidence, I say Texas' lack of interstate connections was not a primary cause of this event.

 

[Astronuc]

A giant asteroid crashing into Earth could cause a world-wide blackout. But we wouldn't care, because we would all be dead anyhow.

Reminds me of a question concerning the safety of nuclear power plants from an anti-nuclear activist, "What would happen if a giant asteroid hit a nuclear plant?" The paraphrased response was "if an asteroid hit a nuclear plant, the plant wouldn't be the problem."

With respect to the FOX news article: "The price hikes affected only those customers on variable or indexed-rate plans, not those with a fixed-rate." Don't pick variable or indexed-rate plans, otherwise read the details. I was wondering if one could apply a force majeure provision.

I say Texas' lack of interstate connections was not a primary cause of this event.

Not a primary cause, but certainly a contributor to the fiasco that unfolded. If power had been available from Louisiana and/or Arkansas, assuming there was plenty of reserve/excess, then perhaps gas transmission lines could have delivered gas if the problem had been losing electrical power to the compressors.

As for gas lines freezing, I understand that it was instrumentation, or perhaps moisture in the gas, rather than the gas itself that froze. Melting point of methane: -295.6°F (-182°C). LPG is a blend of propane and butane, and perhaps small amount of isobutane, butylene, propylene. LPG (propane) has -42°C boiling point, -188°C freezing point.

 

[Astronuc]

With respect to gas lines freezing,

Natural gas wells and pipes ill-equipped for cold weather are a big reason why millions of Texans lost power during frigid temperatures this week. As temperatures dropped to record lows across some parts of the state, liquid inside wells, pipes, and valves froze solid.

Ice can block gas flow, clogging pipes. It’s a phenomenon called a “freeze-off” that disrupts gas production across the US every winter. But freeze-offs can have outsized effects in Texas, as we’ve seen this week. The state is a huge natural gas producer — and it doesn’t usually have to deal with such cold weather.

https://www.theverge.com/2021/2/17/22287130/texas-natural-gas-production-power-outages-frozen

While the frigid cold slashed fuel supplies of all sorts, it also drove up demand for natural gas to heat homes. That “mismatch” is what’s driving these blackouts, says Coombs. There simply hasn’t been enough fuel on hand to power the state’s electricity needs. Natural gas production was pretty much halved in Texas and its gas-rich Permian Basin during the recent cold and stormy weather. It fell from 22.5 billion cubic feet of gas produced per day in December to between 10 to 12 billion cubic feet of gas per day this week, according to estimates from BTU Analytics.

That drop-off in production is thanks to freeze-offs at wellheads where oil and gas are pumped out of the ground. But the cold has also stopped equipment from working properly at gas processing plants, Coombs says. Processing plants separate gas from fluid and impurities; when equipment freezes, plants have to heat it up or wait for temperatures to rise before they can resume their work.

While other states invest more in equipment that helps prevent freeze-offs, Texas hasn’t seen the need. North Dakota typically sees 20 days a year with freeze-off events, while the Permian Basin would normally have just four days a year with freeze-offs disrupting gas production, according to BTU Analytics.

Production of natural gas in the state has plunged, making it difficult for power plants to get the fuel necessary to run the plants. Natural gas power plants usually don’t have very much fuel storage on site, experts said. Instead, the plants rely on the constant flow of natural gas from pipelines that run across the state from areas like the Permian Basin in West Texas to major demand centers like Houston and Dallas.
. . .
The systems that get gas from the Earth aren’t properly built for cold weather. Operators in West Texas’ Permian Basin, one of the most productive oil fields in the world, are particularly struggling to bring natural gas to the surface, analysts said, as cold weather and snow close wells or cause power outages that prevent pumping the fossil fuels from the ground.

“Gathering lines freeze, and the wells get so cold that they can’t produce,” said Parker Fawcett, a natural gas analyst for S&P Global Platts. “And pumps use electricity, so they’re not even able to lift that gas and liquid, because there’s no power to produce.”

https://www.dallasnews.com/news/wea...or-power-it-wasnt-ready-for-the-extreme-cold/
It's going to be up to Texans to sort out this mess.

Similarly, with wind turbines, they can operate (as long as there is sufficient wind) in cold climates without freezing.

 

[anorlunda]

Similarly, with wind turbines, they can operate (as long as there is sufficient wind) in cold climates without freezing.

It's not the temperature per se, but the accumulation of ice on the blades, which cause imbalance.Regarding contributory causes. Almost all of us (including me) want one or two primary things to blame for bad outcomes. A hypothetical event with 20 causes, each contributing 5% to the risk, would be profoundly unsatisfying to people who would like to render judgement.

 

[Astronuc]

Nuclear plants aren't necessarily immune to faults:

On Monday, Feb. 15, 2021, at 0537, an automatic reactor trip occurred at South Texas Project in Unit 1. The trip resulted from a loss of feedwater attributed to a cold weather-related failure of a pressure sensing lines to the feedwater pumps, causing a false signal, which in turn, caused the feedwater pump to trip. This event occurred in the secondary side of the plant (non-nuclear part of the unit). The reactor trip was a result of the feedwater pump trips. The primary side of the plant (nuclear side) is safe and secured.

https://atomicinsights.com/south-texas-project-unit-1-tripped-at-0537-on-feb-15-2021/
https://www.nrc.gov/reading-rm/doc-collections/event-status/event/2021/20210216en.html#en55104

 

[Astronuc]

Almost all of us (including me) want one or two primary things to blame for bad outcomes. A hypothetical event with 20 causes, each contributing 5% to the risk, would be profoundly unsatisfying to people who would like to render judgement.

I like complicated problems. I've done plenty of root cause failure analyses, mostly with respect to nuclear fuel and core component failures, and there are often several factors (e.g., design, materials, manufacturing and operation). Often though, it's a matter of what was overlooked or ignored in design and/or operations.

 

[PeroK]

It's not the temperature per se, but the accumulation of ice on the blades, which cause imbalance.Regarding contributory causes. Almost all of us (including me) want one or two primary things to blame for bad outcomes. A hypothetical event with 20 causes, each contributing 5% to the risk, would be profoundly unsatisfying to people who would like to render judgement.

It's not a question of rendering judgement, but that the widespread extent and prolonged nature of the failure cannot be attributed to one technical fault. Almost by definition, any failure on this scale must ultimately point to a systematic failure of governance. This is well beyond what could possibly have been caused by one engineering or technical oversight - or even several technical blunders.

Ultimately, the root causes of something like this must be financial and political in nature. The technical failures, whatever they are, are ultimately a manifestation of a failure of governance of the state's power and water supplies.

There can be no doubt about where the accountability lies.

 

[artis]

Well I think this is mainly the same problem as with the joke about the "Jamaican hockey team", it's definitely not a new or unheard of event that people tend to get used to the conditions that prevail in the long term and considering a once in a century or once in a half century type of event is unlikely.
I don't want to sound anti free market but I think investors and private companies are also largely to blame here and maybe even understandably because who would want to equip everything for a -20 C winter when there simply isn't one.
It is definitely not a matter of technical incapability, Russia for example has a huge natural gas infrastructure of which at least half is located in permanent freezing like the Yamal gas field and processing plant.
I know a dozen of reactors off the top of my head that have worked or work in winter conditions so again should be no problems there.I think it's simply not having enough redundancy and the cause for that well I think we all know, is most likely the wish to earn the same dollars without having to invest more.

Maybe we need a new patent for wind turbine blades with heating for turbines that work in winter conditions. Just build in some resistive elements and attach a two wire slipring to the shaft, one could simply use the power generated right on the spot with the help of a temp sensor and some electronics to switch on or off, depending on weather conditions.


It seems like a rather long and exhausting process.

 

[PeroK]

Well I think this is mainly the same problem as with the joke about the "Jamaican hockey team",

Do you mean the Jamaican bobsleigh team?

 

[anorlunda]

Maybe we need a new patent for wind turbine blades with heating for turbines that work in winter conditions.

You're trivializing a problem that's more difficult than you think. Studies in many countries have investigated that question and looked at many methods in addition to heating elements. Try this search:
https://duckduckgo.com/?q=wind+turbine+throwing+ice&ia=web

Also, consider this:

For safety reasons, the turbines are shut down while the heating elements melt off the ice, Kurt said. That way, there’s no chance of ice flying off spinning blades, potentially damaging the turbines or, worse, striking someone on the ground, she said.

 

[Astronuc]

"Rolling blackouts" are supposed to be designed so that no customer is out for more than 3-4 hours (they said 12 hours in the WSJ article). Even in thin walled houses, it may take longer than that for pipes to freeze. But this week, some customers were out for more than 72 hours. That indicates a second level failure. The rolling blackouts didn't roll.

This article in the Texas Tribune shows an outage map of Texas from 1000-1100 on February 16.
https://www.texastribune.org/2021/02/16/natural-gas-power-storm/

It's interesting how non-uniform it was, and how some went without power for a few hours to a few days, while other didn't lose power (not referring to El Paso).

I checked in with another family member. She stated they lost power, but unlike many others, they have a backup generator.

 

[anorlunda]

It's interesting how non-uniform it was, and how some went without power for a few hours to a few days, while other didn't lose power (not referring to El Paso).

Thanks for the hard data @Astronuc . Imposition of rolling blackouts is a manual thing. People decide where and when, and the influence of politics and/or personal preference are always suspected. At my organization during the 2003 blackout, one operator said "We are about to loose S*******," referring to an upstate city with population 150K. The supervisor said, "F*** S*******. Protect New York City." It has always been the case, that in New York State, only New York City counts.

 

[Astronuc]

At my organization during the 2003 blackout, one operator said "We are about to loose S*******," referring to an upstate city with population 150K. The supervisor said, "F*** S*******. Protect New York City." It has always been the case, that in New York State, only New York City counts.

Oh, I remember that day. Thanks to deregulation, the local utility, like other utilities, had sold off their plants to merchant producers, so they had to buy power from the grid. Prior to deregulation, the local utility could have isolated itself and remained up. After deregulation, when the grid goes down, everyone goes down. Deregulation never reduced cost or save consumers money, as there was never any significant competition (but that is a different matter).

I was sitting at my desk in my office, which overlooked Main Street in a small city. The lights dimmed, flickered, stabilized, then flickered, went off, came back on, then went off again. I thought at the time, that the construction crew working outside had hit another buried power cable, which had happened earlier that week (the whole street had been dug up to convert a pedestrian section into an actual street for vehicles). Power was completely out. I thought it was the blocks or blocks, but then I called my wife to tell her I would probably be home before the usual time, and she told me that the house and neighborhood had no power. Then we started hearing from others that they had no power, and it was not only the city and our area, but regional, and statewide, and ultimately regional up into Ontario. I hurried home and immediately went to a local convenience store to grab some ice.

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

As part of my graduate course work, I had taken a class in network analysis and reliability. I had expected the local system to be better protected.

 

[Office_Shredder]

Thanks for the hard data @Astronuc . Imposition of rolling blackouts is a manual thing. People decide where and when, and the influence of politics and/or personal preference are always suspected. At my organization during the 2003 blackout, one operator said "We are about to loose S*******," referring to an upstate city with population 150K. The supervisor said, "F*** S*******. Protect New York City." It has always been the case, that in New York State, only New York City counts.

Didn't new york city lose power for like, two days then? Am I thinking of a different blackout?

 

[anorlunda]

Didn't new york city lose power for like, two days then? Am I thinking of a different blackout?

No, NYC was out from about 4:15PM to 11PM the same day.

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#New_York

 

[Office_Shredder]

No, NYC was out from about 4:15PM to 11PM the same day.

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003#New_York

The only mention of 11pm I see there is that the path resumed operating.

In fact,

https://www.nytimes.com/2003/08/15/...blacks-northeast-hitting-cities-8-states.html

Very primary source that specifically states Manhattan did not have power back at 11pm.

https://www.nytimes.com/2003/08/15/...blacks-northeast-hitting-cities-8-states.html

Says power was off in most of the city for about a day (though it doesn't really say when on the 15th it came back).

https://www.nytimes.com/2003/08/15/...blacks-northeast-hitting-cities-8-states.html

Picture 26 has a person on the 15th charging his laptop off a car battery in broad daylight. Picture 28 also describes the second day of the blackout.

I think substantial parts of the city had no power at least for the morning of the second day.

On the other hand I can find literally no sources from any time that say anything about how long syracuse didn't have power for, so your main point still stands.

 

[anorlunda]

The only mention of 11pm I see there is that the path resumed operating.

You're right. The thing I read on the wiki was only about the subway. But I missed this.

Most places restored power by midnight (within 7 hours), some as early as 6 p.m. on August 14 (within 2 hours).[2] New York subways resumed limited services around 8 p.m.[2] Full power was restored to New York City and Toronto on August 16.

So yes, a 2 day outage for many people in NYC.

 

[jedishrfu]

View attachment 278161

Reminds me of when we first came to Texas, we went to a McDonalds and asked for tea.

The server started to put ice in the cup and we said no no we wanted regular tea. She then proceeded to just put in the ice tea mix. We didn't know if she was about to put it in the microwave but a manager stepped in quick, apologized, said he was from NY and then said sadly we don't sell hot tea here.

I've also heard the counter story of a Texan in NYC asking for tea and getting hot tea.

 

[OCR]

.
Oh my. . . !

Sequence of events...

12:15 p.m... An operator corrects the telemetry problem, but forgets to restart the monitoring tool.

Or, to put it in other words. . . .

.