Sunday, May 9, 2010

The Piping Diagrams

Late last month, the legal investigative report on Vermont Yankee was released by the Public Service Board. Report of Investigation Entergy Vermont Yankee was written by the law firm of Morgan, Lewis & Bockius. To prepare the report, the firm spent 2,300 hours interviewing 29 people and reviewing 65,000 documents.

The Missing Piping Diagrams

In an early February post, I wrote: A huge portion of this story is missing. In all this testimony, nobody asks for the piping drawings. People send emails or ask questions: "Do you have such piping?" They get answers: "Nope." Nobody ever asks to see a diagram.

I also pointed out that if the regulators and panels didn't request the information, or if they didn't examine the information they had requested, there is plenty of egg-on-the-face to go around.

I Found the Diagrams

On page 25 of the pdf (page 19 of the text) of the Report of Investigation, I found the following information, which I have paraphrased for clarity:
In a meeting with the NSA team on September 9, 2008, Entergy provided the team with the Piping and Instrumentation (P & ID) diagrams of the Main Transformer, HPCl, Condensate & Feedwater, Residual Heat Removal, Cooling Tower, Service Water, and Condensate Storage with underground piping.
On page 27 and 28 of the pdf (page 21 and 22 of the text) there is a description of a further meeting on September 11, 2008, in which VY gave NSA tables of piping that was included in VYs Buried Piping and Tanks Inspection and Monitoring Systems, (BPTIP) reviews. The tables included the AOG piping (which was later the cause of the problem) and other systems associated with the tritium review program. Contamination levels were described for each length of pipe.

One of the VY people (Naeck) said (page 22, paraphrased) that the NSA people were smart engineers. He felt NSA knew of the existence of piping that carried radionucleides.

Who knew?

NSA knew, as above.

Who else knew? The NRC knew. At the NRC meeting on April 19 in Brattleboro, NRC staff members stated that they had records of piping at Vermont Yankee. They stated that they did not come forward during the NSA or POP review process because the NRC does not participate in hearings between a plant and its state regulators. However, I am reasonably sure that if a state regulator asked them for piping diagrams, they would have agreed to share them. They might have had to get permission from the power plant. I don't know. Of course, nobody asked the NRC because the NSA already had the diagrams.

In short, everyone knew.

PowerPoint or Piping Diagrams?

Recently, Donald Kreis of the Vermont Law School has written about this September 11 meeting which included the tables of piping. However, he describes the meeting as being based on a misleading PowerPoint presentation. Kreis ignores the piping tables which were given to the NSA at that meeting.

Like Professor Kreis, I am a fan of Edward Tufte. Like Tufte, I have mixed feelings about the use of PowerPoint. PowerPoint certainly played a role in the shuttle disaster Kreis mentions, but the corporate culture of NASA played a bigger role.

I do appreciate Kreis use of effective rhetoric. He puts VY, shuttle disasters, and Afghanistan losses together in one article, tarring VY with completely unrelated disasters. This was a true tour-de-force, and worthy of a lawyer who knows how to persuade people. I can't write a comment on his article, though, because comments are closed. Perhaps Kreis will comment on my post instead? I think it would be worthwhile to have a dialog.

After the VY-NSA Meeting

Let's leave Kreis and PowerPoint now, and go back to the Report of the Investigation. The early pages show VY and the NSA amicably deciding that the Service Water System (SWS) is a more vulnerable system that the CST (Condensate Storage System) or other systems. They mutually conclude that piping in trenches is likely to be far less of a problem than the SWS, and they decide to investigate the SWS.

As a former corrosion engineer in nuclear plants, I agree with this decision to switch the focus from the CST to SWS. The SWS uses untreated water for cooling the condenser, and always poses problems of corrosion and biofouling. As one engineer at VY says early in the report Breite, quoted on page 25 of the pdf, 19 of the report: (I was) not surprised (we were looking at SWS) because everyone looks at SWS. SWS across the industry consistently raised reliability concerns.

Much later, things got difficult. The players changed and the questions changed. The agreements about what systems to investigate did not hold, but that is another post for another day.

Right now, we can leave the situation as amicable as it was in late September 2008. Piping diagrams were available. NSA and Entergy had agreed to investigate SWS. Groups were in agreement. (We're now on page 40 0f the pdf and 34 of the report.)

Later, things changed.

Beyond Egg

In recent articles, DPS and Arnie Gundersen claim that they were not given sufficient information. Since they had access to P & ID diagrams and pipe lists, what information did they feel they were lacking? (Perhaps they are just trying to rub the egg off their faces. Or perhaps information was lacking, though the articles don't say what information they were denied. )

I recommend Rod Adam's excellent post on this subject. Adams gives important insight into people's motivations.

I also plan to get beyond page 40 of this 140 page report. Later. In another post.

Graphic of part of a P&ID diagram in accordance with Creative Commons license.


Don Kreis said...


Thanks for the invitation to leap into the dialogue. If I am understanding our respective comments correctly, we are a couple of Tufte fans who aren't in fundamental disagreement here. Diagrams and technical reports would have been a good focus at the September 11, 2008 meeting, which instead was organized around a .ppt presentation that was unhelpful for precisely the same reason that the Boeing .ppt show was unhelpful during that final flight of the Shuttle Columbia.

Obviously, the 9/11/08 PowerPoint didn't have the devastating consequences the Boeing/NASA one did. I never suggested otherwise. Your real beef, I think, is with Tufte -- he could have chosen other, less high-stakes examples of bad PowerPoint slides. The world is awash in them.

David O'Brien, Commissioner said...

Meredith, with all due respect I have to disagree with your characterization of the Department having egg on our face with respect to the now famous underground pipes. When others are looking at this issue they are missing a critical piece that is not in the VY investigation or recorded even in our emails. It is the repeated claims by VY persons at the many meetings we had together with the consulatants and the oversight panel where VY was questioned on their claim that they did not have any underground buried pipes. It was this repeated asseration by VY that we trusted that they were giving us an accurate answer.

There are so many people saying why not just look at the drawings. Well, first, they probably have no idea how many drawings there are for this plant and the drawings alone do not identify pipes carrying radionuclides. After discovering that the company's own affidavit by a Mr. Rademacher could not be relied upon, Bruce Hinckley, our consultant that assisted us with the NSA audit, had to have contractors go over the drawings and other information to identify the pipes that do carry radionulclides and it took about 5 weeks and cost about a 150K. People forget that the legislation had us on a very tight schedule where we had less than 4 months to complete a massive inspection. We had to get the scope of the inspection agreed upon with the POP so we could get started before the clock ran out. We did not have reason to believe that VY was not providing accurate information with regard to the pipes and therefore had no reason to second guess them by spending more time and money to verify their statement upfront.

In summary:

1. First, the question isn’t just about whether there were underground pipes at VY. Of course, there are underground pipes at VY. We all knew there were underground pipes at VY, but what we were told is that there were no underground piping systems carrying radionuclides. So we could have been aware of numerous drawings showing underground pipes and that would not have alarmed us or clued us in that there are underground pipes carrying radionuclides.

2. Second, as a long time regulator, we have to trust a utility to provide us with accurate information. Our entire system of regulation is based upon that basic premise. If that was not true, the DPS and every other PUC in the country would have to be at least twice as large. Once a utility shows it cannot be trusted to give accurate information, then we have to go over everything with a fine toothed comb. We said we would verify the piping affidavits that Entergy gave us because we no longer trusted them to give us correct information. Hence the extra 5 weeks and 150K worth of verification.

Meredith Angwin said...

First, I would like to thank Donald Kreis and Commissioner O'Brien for their thoughtful responses to my blog.

Donald, just a note that it is not possible for me be annoyed at Tufte. I have admired him since I first saw his first book, The Visual Display of Quantitative Information. And he can use whatever examples he chooses.

My issue was different. The Report of the Investigation clearly states that Entergy gave P and ID diagrams and tables of piping to the NSA team in meetings on September 9 and 11, 2008. As I understand it, the NSA team received this piping information in both hard copy and electronic form. (This is not in the report, it is something I heard through the grapevine. Maybe NSA actually only received paper copies, I don't know for sure.)

You concentrate on the Powerpoint presentation, and ignore the backup data which Entergy gave to the NSA participants. In my post, I cited the pages for my sources of data exchange. Perhaps you can read pages 19 through 23 of the text (25 through 30 of the pdf) and tell me why you think the Powerpoint was the only significant information-exchange method?

Commissioner O'Brien, I want to thank you for your response, which was full of new information (or at least, new to me). I didn't understand the legislative requirement to complete the massive inspection in four months. I had no idea of the constraints under which the Department was operating. From an outsiders point of view, the whole thing has basically gone on forever, with dockets, VSNAP meetings, NSA panels, POP panels, etc. When you are working inside the project, though, life looks different. The Department was clearly put in a position of Never Time to Do It Right, Always Time to Do It Over. Six weeks and 150K to do it over! Thank you for explaining the situation.

Thank you both for your comments.