San Onofre steam generator tubes leaking - why?

[gmax137]

Combustion Engineering's IP are owned by Westinghouse

true enough

...since ABB merged Westinghouse Nuclear Services and CE together.

Not quite. ABB never had any part of Westinghouse.

In 1990, ABB bought Combustion Engineering (both the nuclear and fossil power businesses).

In 2000, ABB sold their nuclear businesses to Westinghouse Electric. Shortly after that, they sold their fossil equipment business (including the CE fossil interest) to Alstom.

WEC is the nuclear part of the "old" Westinghouse, which had been sold to BNFL when CBS broke the old Westinghouse into fragments around 1996. So if anything, BNFL is the one that "merged" CE nuclear into WEC (but even that is stretching things).

In 2006, BNFL sold Westinghouse Electric (including the previously absorbed CE nuclear) to Toshiba.

Interestingly, in the past Westinghouse and MHI had a good relationship (MHI built the PWRs in Japan originally following the W design under license). When BNFL sold Westinghouse, MHI was in the bidding, but Toshiba won out.

Anyway, Toshiba now owns the Westinghouse nuclear business, including the CE technology. Due to the Toshiba / MHI competitive dynamic it was never likely that they would cooperate on the SONGS RSG design.

 

[jim hardy]

Ouch! So much for the Renaissance.

Crystal River is also down and out, and Kewaunee was just permantely shutdown. So the number of operating reactors has dropped to 100, down from 104.

There's a trickle of new construction here in US:

http://www.nytimes.com/2013/06/12/b...oject.html?pagewanted=1&_r=1&ref=matthewlwald

Until recently, a new reactor construction project had not been started in the United States for 30 years, and now Vogtle and a similar project in South Carolina, V.C. Summer 2 and 3, are supposed to provide the answer to nuclear power’s great questions: What does a new reactor cost? With the price of natural gas near historical lows, can it even be worthwhile?

As the current generation of reactors moves toward retirement, the two projects may be the industry’s last best hope.

“Everybody’s watching the construction of that plant,” said Barry Moline, executive director of the Florida Municipal Electric Association, speaking of Vogtle. Several association members are considering investing in a nearly identical plant proposed by Florida Power and Light in Miami. Mr. Moline said of Vogtle’s builders, led by Georgia Power, “If they can do it, that will be the model.”

And if they can’t, it could be years before anybody thinks of trying again. The new designs are supposed to be a tenth as likely to have an accident and to be easier to operate, but if they cannot be built roughly on time and on budget, then nuclear power will have trouble in the era of plentiful natural gas and emerging technologies like wind.

Nuclear power could become a bypassed technology — like moon landings, Polaroid photos and cassette tapes.

Bypassed technology indeed.
I worked on the Westinghouse analog control systems. I feel sort of like a Brontosaurus veterinarian .
Well, it was interesting while it lasted.

And that carbon tax could start it all over again.

old jim

 

[atyy]

but IMHO the old bureaucratic dodge of "just blame the vendor" isn't honest.
SCE needed to provide Mitsubishi any help they needed, even if they had to pay CE for some trade secrets and Westinghouse for some consulting.

How would that happen? Mitsubishi would have to know they were incompetent and ask SCE for help? But did Mitsubishi know their analysis was inadequate before the fact (should they have known, could they have known?)

 

[jim hardy]

How would that happen? Mitsubishi would have to know they were incompetent and ask SCE for help? But did Mitsubishi know their analysis was inadequate before the fact (should they have known, could they have known?)

I am sure that had Mitsubishi known they would have refused to build the generators.

Here is my reasoning for not excusing the utility.

Any design change is the sort of thing a prudent engineer triple checks. On something this important you quadruple check and then get somebody else to check your work.

Here's what they DID know:

SCE's engineers know that Mitsubishi long ago licensed from Westinghouse to build reactor components. So Mitsubishi is unquestionably qualified to build a replacement for a Westinghouse steam generator.

SCE's engineers also know that being a CE plant, San Onofre's steam generators differ in design by whatever CE and Westinghouse did to avoid one another's patents.

SCE's engineers also know that they made a design change to smaller diameter tubes.
Skinny tubes are more limber than fat ones.

SCE's engineers also know that due to different "Recirculation Ratio" of the Westinghouse and CE designs, the fluid properties in the u-bend region differ. Mitsubishi is accustomed to building for the Westinghouse properties.
(If I recall correctly the respective recirc ratios are 10 and 4.)

In my opinion those two italicized facts should have been a red flag to SCE's engineers that they must pay very special attention to that region.

Apparently they had an inkling:
from that NEI article:

AVB support structure
The term ’AVB structure’ describes tube
supports in the tube bundle U-bend region.
The AVB structure had to be designed such
that the potential for tube wear due to flow
induced vibration was minimized.
To achieve this objective, six sets of Vshaped
AVBs made from Type 405 ferritic
stainless steel, providing up to 12 support
points per tube bend, were installed in the U
bend region to provide support in the region
where the tubes are most susceptible to
degradation due to wear from flow-induced
vibration. The single major challenge here
was control of the AVB thickness and
flatness, and tube-to-AVB gap size. This
challenge was addressed by customizing the
fabrication and assembly processes and
implementing strict quality control in various
stages of AVB fabrication and AVB structure
assembly.

Skinnying-down the tubes made a fundamental change to the mechanical properties of the tubes, one that affects their dynamics.

Had that NEI article mentioned " Because of the mechanical changes to the tubes and the potential effect dynamic behavior, an independent review was performed by third party"
I would have more sympathy for the utility.

But from what I've seen, they accepted Mitsubishi's analysis.

And that u-bend area is right where they had their trouble.
From Nuceng's link http://www.nrc.gov/info-finder/reactor/songs/ML12188A748.pdf

If operating velocities reach this critical value, vibration amplitudes can increase rapidly and fluid-elastic instability forces can lead to rapid pulsation and damaging of tubes. The U-bend region is most susceptible because (1) the local fluid has a higher void fraction, with high velocities; (2) the fluid flow is in a direction normal to the tube, and (3) the anti-vibration bars are limited in their dampening capability along the plane of the tubes. Traditional design of anti-vibration bar systems have not considered in-plane fluid forces since it was accepted that the rigidity and dampening strength of the tube in this direction was adequate to preclude it. This event at SONGS is the first US operating fleet experience of in-plane fluid-elastic instability, sufficient to cause tube-to-tube contact and wear in the U-bend region. ). The parameter of local velocity divided by critical velocity is referred to as stability ratio. The accuracy of calculating fluid-elastic instability is limited based on inputs that are best determined by design-specific mockup test data. Mitsubishi did not perform design-specific mockup tests, but used generally accepted test data, and other data based on Mitsubishi test rigs that were not specific to the SONGS replacement steam generator design.
The team noted that Design Specification SO23-617-1 did not address specific criteria for stability ratio and does not mention fluid-elastic instability. The team did find that the Mitsubishi calculated design values for stability ratios did not exceed 0.5. It is important to note, that each steam generator manufacturer has different design values for maximum stability ratios; therefore there is no standard value. The smaller that the design stability ratio is (has to be less than 1), the more margin to fluid-elastic instability.

Bold suggests the SCE procurement spec didn't even address fluid stability.

When you push a vendor too hard you take the risk he'll make a mistake.
An astute manager knows it's necessary to be a helpful partner, even if you have to give up appearing such a tough customer.So how could they know?
They couldn't.
IMHO there was good reason to suspect - So they needed to make absolutely sure.
It became a matter of judgment and somebody goofed.
old jim

 

[gmax137]

How would that happen? ...

I agree that this would have been very difficult for practical reasons if nothing else (see my post above). But recognizing that the CE steam generators are quite different (twice the size) of the typical MHI SGs, it might have been wise of SCE to put some kind of compensatory effort in place (like a third-party review) once they decided to go with a designer other than the OEM. Who knows - maybe that happened and we haven't heard about it. The bid evaluations that led to SCE selecting the MHI offer might be interesting.

edit: cross posted with Old Jim's post above. I think we're on the same wavelength

 

[etudiant]

Not to pile on, but in this instance everyone lost by trying to avoid outside scrutiny and review.
So the industry as a whole suffers another avoidable black eye and 1100 people lose their jobs.
If the regulators had not been bypassed in the first place, they would surely have been sufficiently invested in the plant to let it run at a safe 70%, even assuming that they would not have caught the design defect.

 

[atyy]

When you push a vendor too hard you take the risk he'll make a mistake.
An astute manager knows it's necessary to be a helpful partner, even if you have to give up appearing such a tough customer.

Do you think they pushed the vendor too hard, or was it just not being careful enough (well, I guess pushing the vendor too hard is also not being careful enough)? Like gmax137 and etudiant maybe leave it open it's more of the latter?

So do the laws really put all the blame on the vendor, when as you argue, it should also fall on the utility? If so, are there modifications to the law that can prevent this - I assume it's not in the state's interest to suddenly run out of power?

 

[jim hardy]

Do you think they pushed the vendor too hard, ...

Well, without actually being there there's no knowing what was the dynamic between the two outfits.

The Mitsubishi non-proprietary 'root cause' report at
http://libcloud.s3.amazonaws.com/93/92/5/2795/MHI_Root_Cause.pdf
has a couple interesting paragraphs in it:

The Certified Design Specification S023-617-01, Rev. 3 stated that SCE intended to use the provisions of 10 C.F.R. §50.59 as the justification for the RSG design, which imposed physical and other constraints on the characteristics of the RSG design in order to assure compliance with that regulation.

I don't know what were the constraints.

Mitsubishi built steam generators for Ft Calhoun , pdf page 9 of 64

The replacement steam generators for OPPD's Fort Calhoun Nuclear Generating
Station are the only other steam generators designed by MHI operating in the United
States. The OPPD RSGs replaced Combustion Engineering OSGs and are of a similar
design and construction as the SONGS RSGs with certain differences, including:
" Identical tube diameter (3/4") and wall thickness (0.043")
" Identical tube pitch (1.0" equilateral triangle)
" Identical pitch-to-diameter ratio (P/D = 1.33)
" OPPD has greater average tube to AVB gap
* OPPD RSGs are smaller than SONGS RSGs
* Fewer AVBs than SONGS
" Fewer tubes than SONGS
" Smaller U-bend radius than SONGS
" Lower maximum steam quality (void fraction) than SONGS

The Fort Calhoun RSGs have operated more than three fuel cycles with no evidence of
U-bend tube degradation (no tube-to-AVB wear, no tube-to-tube wear, and no
retainer bar-to-tube wear).Other steam generators designed by MHI (operating
outside of the United States)are of a different design and have a variety of tube sizes,
tube pitches and operating conditions. These steam generators have years of
operation without significant tube wear.

So, when they stuck to original CE physical design they were okay.

But they tinkered with the design:
Page 48 of 64 :

Identification of the changes from previous SG designs led to the recognition that the RSG design deserved close scrutiny. MHI considered the changes in the SONGS design from previous steam generator designs and compared the basic design parameters of the SONGs RSGs (e.g., heat transfer area, circulation ratio, steam pressure, etc.) with other steam generator designs. Further, as part of the development of the SONGS RSG design, MHI conducted a detailed comparison between its proposed AVB support for the tubes in the U-bend region and that of a comparison plant of similar design. A special AVB team was formed and included
industry experts to conduct an extensive design review process in 2005 / 2006 to optimize the U-bend design and address the technical issues. The team concluded that the SONGS design was significantly more conservative than previous designs in addressing U-bend tube vibration and wear. Also MHI and SCE recognized that the SONGS RSG steam quality (void fraction) was high and MHI performed feasibility studies of different methods to decrease it. Several design adjustments were made to reduce the steam quality (void fraction) but the effects were small. Design measures to reduce the steam quality (void fraction) by a greater amount were considered, but these changes had unacceptable consequences and MHI and SCE agreed not to implement them. It was concluded that the final design was optimal based on the overall RSG design requirements and constraints. These included physical and other constraints on the RSG design in order to assure compliance with the provisions of 10 C.F.R. §50.59. (emphasis mine - jh)

So do I think there was a push to avoid that 50.59 trigger ?

It's reading between the lines, but yes I do.
Phrases "close scrutiny"
and "optimal based on the overall RSG design requirements and constraints"
tell me somebody was worried but they yielded to schedule and contract pressures.

old jim

 

[jim hardy]

Interesting presentation on the failures here:

https://ric.nrc-gateway.gov/docs/abstracts/palmisanot-hv-w13.pdf

 

[jim hardy]

oops -
I think I have to withdraw my claim that SCE changed tube diameter.
In process of double checking myself I can not find where I picked up that idea.
They changed the material and number of tubes and heat transfer area,

but I cannot find a statement that they changed diameter. Don't know where I got the impression.

sorry for posting that mis-statement.

old jim

 

[Hiddencamper]

Well, without actually being there there's no knowing what was the dynamic between the two outfits.

The Mitsubishi non-proprietary 'root cause' report at
http://libcloud.s3.amazonaws.com/93/92/5/2795/MHI_Root_Cause.pdf
has a couple interesting paragraphs in it:

I don't know what were the constraints.

Mitsubishi built steam generators for Ft Calhoun , pdf page 9 of 64



So, when they stuck to original CE physical design they were okay.

But they tinkered with the design:
Page 48 of 64 :


So do I think there was a push to avoid that 50.59 trigger ?

It's reading between the lines, but yes I do.
Phrases "close scrutiny"
and "optimal based on the overall RSG design requirements and constraints"
tell me somebody was worried but they yielded to schedule and contract pressures.

old jim

The 50.59 and other things are available here

http://www.nrc.gov/info-finder/reactor/songs/songs2/publ-avail-doc.html


Here is the direct link to the 50.59 http://pbadupws.nrc.gov/docs/ML1305/ML13050A189.pdf

proprietary stuff removed.

More than anything, There's a number of questions they answered "NO" to in the screening for adverse effect that I think are total horsegarbage (based on my experience as a qualified 50.59 engineer). Like the 50.59 screening is 15 pages...that's absurd. If you are using more than 4-5 pages for a screening you are doing it wrong (and it probably SHOULD be in the evaluation).

That said, I think based on what I do know, and the information they had at the time, that the evaluation for most of the changes would have been acceptable (the "no more than a minimal increase" sections).

 

[jim hardy]

The 50.59 and other things are available here



That said, I think based on what I do know, and the information they had at the time, that the evaluation for most of the changes would have been acceptable (the "no more than a minimal increase" sections).

Thanks for the links !

I don't disagree with the premise of like-for-like replacement under 50.59.
Just so it's done well .

For me, the question comes down to "...could anybody have reasonably forseen this vibration problem in the tubes? " .
That other computer codes found it so quickly in the followup investigation suggests to me that somebody might've.
But I'm an instrument maintenance guy not a fluids analyst.From: http://www.nrc.gov/info-finder/reactor/songs/ML12188A748.pdf

Both the NRC and Mitsubishi ATHOS results were reasonably consistent and strongly suggested that high velocities coupled with high void fraction were primary causal factors in the tube fluid-elastic instability and the excessive wear patterns observed in the Unit 3 steam generators.

From Root cause report , pdf page 51 http://pbadupws.nrc.gov/docs/ML1306/ML13065A097.pdf

Also MHI and SCE recognized that the SONGS RSG steam quality (void fraction) was
high and MHI performed feasibility studies of different methods to decrease it.
Several design adjustments were made to reduce the steam quality (void fraction)
but the effects were small. Design measures to reduce the steam quality (void
fraction) by a greater amount were considered, but these changes had
unacceptable consequences and MHI and SCE agreed not to implement them. It
was concluded that the final design was optimal based on the overall RSG design
requirements and constraints. These included physical and other constraints on the
RSG design in order to assure compliance with the provisions of 10 C.F.R. §50.59.
(underline mine- that was mid 2005 per http://pbadupws.nrc.gov/docs/ML1305/ML13057A014.pdf page 56 - jh)

Somebody was worried enough in '05 to want to reduce void fraction..
My guess is the utility wouldn't accept the changes it would have taken to accomplish that. This is how technology progresses, by building ever bigger machines, pushing designs, and seeing what goes wrong with them.
We low echelon guys have a saying "Murphy was an optimist"

Management science progresses similarly I suppose with their ever bigger organizations.

My old mentor used to say "Your evaluation is dandy but that power plant can't read."
I guess that's why the small things of the Earth still occasionally confound the mighty.
Who'd have ever thought the inside of a boiler could be too dry?

old jim

 

[Thermalne]

true enough



Not quite. ABB never had any part of Westinghouse.

In 1990, ABB bought Combustion Engineering (both the nuclear and fossil power businesses).

In 2000, ABB sold their nuclear businesses to Westinghouse Electric. Shortly after that, they sold their fossil equipment business (including the CE fossil interest) to Alstom.

WEC is the nuclear part of the "old" Westinghouse, which had been sold to BNFL when CBS broke the old Westinghouse into fragments around 1996. So if anything, BNFL is the one that "merged" CE nuclear into WEC (but even that is stretching things).

In 2006, BNFL sold Westinghouse Electric (including the previously absorbed CE nuclear) to Toshiba.

Interestingly, in the past Westinghouse and MHI had a good relationship (MHI built the PWRs in Japan originally following the W design under license). When BNFL sold Westinghouse, MHI was in the bidding, but Toshiba won out.

Anyway, Toshiba now owns the Westinghouse nuclear business, including the CE technology. Due to the Toshiba / MHI competitive dynamic it was never likely that they would cooperate on the SONGS RSG design.

Thank you! I did not know that!

 

[atyy]

The Economist has an item on this: http://www.economist.com/blogs/babbage/2013/06/nuclear-power .