Chapter 6. 1970s Vintage Operator Training
When I got out of the Navy in 1970 I returned to my hometown in Ohio. I had been a “Navy Nuke” completing a six year enlistment as a MM1(SS), including a two year stint as a Navy “prototype” instructor at S1C training facility. I was retained there as an instructor after my initial qualification as an Operator. After that instructor tour I was assigned to a new construction Navy submarine, late in the project but before final testing such as Hot Functional Testing, initial criticality, and sea trials. The Toledo Edison Company had just started the initial construction of its Davis Besse Nuclear Plant in 1969, almost in my backyard 10 miles away. I needed a job, applied, and got hired. I was the second Navy nuke they hired, the first being a nuke submarine qualified officer who was slated to be the Operations Manager. Toledo Edison was gearing up to staff Davis Besse, thus they were hiring a lot of new potential Operators, including a lot of Navy nukes. They put us to work in one of the fossil generating stations until it was time to start our formal nuke training.
Regulatory Background
Some minimum requirements that were used by the OLB in determining the adequacy of initial training programs are briefly summarized in the July 30, 1979 memo from Harold Denton to the NRC Commissioners (SECY 79-330E). For applicants for cold examinations a four phase training program is described, which includes:
1 . Twelve weeks of training in basic reactor fundamentals and at least ten startups of a research reactor;
2. Six weeks of lectures on the design features of the specific plant;
3. One to two months of observation of the day-to-day operation of a nuclear power plant and two to three months of operation of a nuclear power plant simulator;
4. One year of on-the- job-training and classroom study including construction check-out activities and preoperational testing.
Our formal nuke training started in about 1972 in a classroom setting. Our whole planned training program was very similar to any new single nuke unit program of that time. With the possible exception of there were plants that had their own Training Simulator at that time. They pulled all the designated nuke plant staff out of the plants and started us in training very similar to Navy nuke classroom training; consisting of math, physics, heat transfer, fluid flow, reactor kinetics, electrical theory, etc. Our instructors were mostly from a professional nuke training company and were all certified as Senior Reactor Operators on US Army Nuke program plants, and they varied in discipline expertise. Our reactor kinetics instructor was a Dr degreed nuclear college professor. This training was about 12 weeks in duration. It was followed by 2 weeks at the University of Michigan pool reactor to actually apply reactor theory to reactor operation doing reactor startups, inverse multiplication plots, etc. We even encountered some Xenon transient problems as the pool reactor fuel would “poison” with Xenon during operation; which we dealt with by moving the poisoned assembly out of the core and replacing it with a fresher assembly.
At the completion of that training we returned to the classroom setting and our training continued for about a month using B&W instructors. This phase consisted of equipment training on the components within the B&W scope of supply for the plant. These B&W instructors were all “experts” in their particular hardware area. The one hardware exception was in the plant Accident Analysis area where our instructor was actually the person who had run the Accident Analysis for the plant Safety Analysis Report that was used for the plant licensing submittal. This accident analysis training was as close as you could get to any actual “plant transient response” training, as there was just nothing else available at the time. There were just not that many plants operating at that time, and there certainly was no “Operating Experience” feedback program out there that could have discussed actual events at operating plants. This Accident Analysis training became so engrained as the actual plant transient response that it was even used on NRC Operator Exams of that era to describe the plant response to an event. They had no relationship to the actual real plant response, mainly due to the “rules” of the analysis, the fact events never cascaded with additional failures, and due to the conservative analysis inputs (everything in the worst direction all at the same time).
At the completion of that classroom training we were divided into two groups for about twelve weeks of “Observation Training.” My group went to the H.B. Robinson three-loop PWR in South Carolina. This training was structured, self guided training. We had check sheets to follow; read system descriptions, procedures, walk down systems in the plant, etc. We also got to watch several actual plant operation activities, mainly testing activities performed by the shift crews. About every two weeks you would have an evaluation visit from a B&W Simulator Instructor who checked your check sheet progress and would give you a plant walk through and ask questions to evaluate your progress.
Shortly after the Observation Training phase we started the three-month B&W Plant Simulator Training at the B&W Lynchburg, VA training facility. My recall is this was 1974-ish. We were getting what's called "cold licenses", meaning the plant isn’t available yet, so the plant can't even load fuel until there is a licensed crew. By this time we were divided up into prospective operating shifts, so we trained as a group with the actual people who would be eventually working together as shift in the plant. At that time Toledo Edison had selected us for our eventual staffing positions. I was the Shift Supervisor, and I had and Assistant Shift Supervisor and two Reactor Operators. We actually rotated position during training so everyone got “hands on” training operating the Control Panel controls. The Simulator training consisted of four hours of classroom and four hours of Simulator operation.
The Simulator training for us was considered to be “generic” training because the Simulator itself was a replica of the Rancho Seco B&W plant control room. The Control Panel controls and instruments were hooked to a big computer. Like the world's most complicated video game. It didn't simply run "movie-like" scenarios but actually had system models and was running thermo-hydraulic calculations to make the meters and such move. It was similar to the idea of airline pilot simulators, where there are some things you just don’t want to do to an actual airplane, never-the-less actual hands on training is necessary for some events. There were substantial hardware differences between Rancho Seco and Davis Besse, but the operating philosophy was similar as they were both B&W supplied plants. The classroom sessions were a repeat of the System Expert sessions we had in Toledo, and also operating sessions done by the B&W Simulator instructors. The B&W Simulator instructors were a mixed bag of experience, but all Senior Reactor Operator (SRO) level. About half had been previously licensed at other commercial PWRs, but not all at B&W plants. The senior instructor was actually an SRO licensed operator at the army reactor in Antarctica, and another instructor was an SRO at the B&W fuel facility in Lynchburg. They were all sharp, competent people but I mention this to point out it was not a large group of very experienced “wise old men” heavy in years of PWR experience, as a reader today might expect.
There were two primary purposes for this training. First, we were a group of brand new Operators for Davis Besse and it was a brand new plant. So the training was totally focused on teaching us to operate that new plant, concentrating on all the aspects of plant operation that entails. Mixed in with normal operations were anticipated casualties or transient events always initiated by a single hardware component failure. These casualty events were obviously more focused towards the latter part of our training when we had increased our proficiency at normal plant operations. So it was just not totally intense casualty training as you might expect a few years later in Requalification Training. The second purpose of the training, and it was never far from your mind, was we were being prepared to pass the required NRC Operator License exam. Without that NRC license, you didn’t have an Operator job, and without a licensed staff Davis Besse wouldn’t run.
The system normal operating procedures and emergency operating procedures we used on the Simulator were not, in fact, real procedures. They were actually Draft Procedures that B&W supplied with the contract for the plant. These procedures used the functional requirements of system operation for procedure steps, and each plant would translate those functional requirements to plant specific procedure steps later at each plant. B&W was training Operators for several plants, and they were all a bit different in design, so this was about the only way that it was possible. Not to mention that we Operators would actually write our plant specific procedures much later at the plant using these functional requirement procedures as guidelines. So Davis Besse plant procedures did not even exist at this time of our Simulator training.
The primary individual plant system learning focus of the simulator training was to learn the complicated Control System of the B&W plant, the Integrated Control System (ICS), or the “brain” that controlled the whole plant. The training was interspersed with casualties, including LOCAs. It also included a few stuck open PORV casualties during which the Pressurizer level indication always went down (from the loss of inventory out the leak). Additionally it included the important training and practical application of using the “legalized” Administrative Requirements, the plant Technical Specifications, to determine if a particular inserted equipment failure required a manual Operator controlled plant shut down.
One thing to keep in mind from an historical perspective was we were getting this training in the mid ‘70s. There was virtually no trading of Operating Experience, and actually not a lot of actual event experience to be had anyway. There were only 60-ish plants even operating. So at that time the transient response understanding (and thus the training) was all based on the “Licensing” Safety Analyses Transients (as explained in Chapter 8). Those were done for a specific purpose, and using specific constrained rules like only a single failure. They had no connection with the real world where simple multiple failures can pile up to create the stinky events. So I’m sure questions will get asked, by newer Operators, totally out of disbelief that things could actually have been that way. But they were. So things that are obvious today were not obvious thirty five years ago.
The Emergency Operating Procedures (EOPs) of that time were written using the plant Final Safety Analysis Report (FSAR) transient response, with one EOP per transient. And that FSAR document was used as the NRC licensing basis for the plant, not an Operators document. Each EOP was only 100% accurate for each analyzed event happening alone. At both DBNPP and TMI we found ourselves in several at once all in the first five minutes (Loss of MFW, Reactor Trip, Turbine Trip, Safety System Actuation, etc), and none of them was 100% accurate for the combination of the total. That system was just flat about unworkable when failures started to combine. To a certain degree you had to pick and choose steps when events combined, and there was no “official” hierarchy to which procedure was the controlling one for what was happening.
Now if you overlay the training for events on top of that transient understanding, it was approached the same way. Since the only transient info available was the Safety Analysis transients, that is how it was taught; a single event using data plots for that event and Immediate Operator Action steps from the EOP for the single event.
So to address questions on how we were trained to handle system saturation, the simple answer is we weren’t, it was not taught for any transient response, because that transient was not specifically analyzed. Since the Pressurizer steam space leak transient was not understood, with the system falling to saturation and just hanging there, the whole thing fell apart when it happened. We had not been trained for that possibility.
But at least in my case, my whole training package experience back, including both navy nuke basic principles training and Davis Besse basis training about steam plants and steam/water properties etc. , did in fact work. It’s what clued me in; I said this can’t be possible (we didn’t pump that water into the pressurizer). So my basic principles training is what helped me figure it out. It was in fact my training and understanding of the basic stuff at the foundation that allowed me to figure out that we were saturated. But still it did not overcome my (and several other folks) conditioning to never pump more water into a full Pressurizer. And believe it or not, I never heard the current Operator term “subcooled margin” used at all, until after TMI.
That’s the historical context, and it can be hard to believe in today’s world I know. Once-upon-a-time it was hard to believe the world was round also. To understand the causes of TMI the whole historical context of what led up to it has to be understood.
During a typical leak drill (speed of all plant responses was dependent on the hole size the instructor had inserted into the model calculation) Pressurizer level (the key indicator) and coolant system pressure always started down, usually resulting in system alarms, etc. and the Operators would start into the Emergency Operating Procedure (EOP) actions. A fairly good size leak would always outrun the initial possible remedial action manual Operator steps, like starting the second system water Make-up Pump, and you'd shortly get an automatic Reactor & Turbine Generator trip (at which time you enter two more emergency procedures, Reactor trip and Turbine trip). Both of these require manual Operator actions including verification all automatic actions did occur. Shortly thereafter you'd hit the low Reactor Coolant System pressure trip that auto started the Emergency Core Cooling System (ECCS) systems (high pressure injection [High Pressure Injection] pumps on, initial phases of Containment isolation, etc). At that point a ton more verifications, making sure both ECCS trains are functioning with all their required cooling support systems operating, etc. By definition a small break loss of coolant accident (SBLOCA) is one where the High Pressure Injection pumps can keep up (by maintaining Pressurizer level). At this point everything is relatively stable, High Pressure Injection is pumping water in, the leak is leaking to the containment floor, but the High Pressure Injection pumps are likely injecting more water than is leaking so the Pressurizer level is starting to go up. It's a rule... never let the Pressurizer level get off scale high, so the Operator starts throttling back the High Pressure Injection flow using the pump discharge valves to maintain Pressurizer level within indicated range. Now you start a system cool down and depressurization, by dumping Steam Generator steam to the Main Condenser; with the goal to get to <250PSI pressure and less than 200F (below boiling point) so you can realign to normal shutdown recirculation cooling mode of the reactor. As the system P & T drop, the lower coolant system back pressure on the High Pressure Injection pump discharge keeps increasing High Pressure Injection flow into the system. This requires constant Operator attention during the cool down, as Pressurizer level starts increasing the operator must constantly throttle High Pressure Injection flow back to keep the indicated level "on scale", along with every other multiple things he's doing to get the plant cooled down.
So here's a pinch point. At the end of our 12 week training we had to take an operating exam on the simulator. B&W supplied the examiner, and an extra simulator instructor to act as your helper, but you took it alone, not with your normal whole shift crew. Such an exam was apparently required by NRC Operator training program regulations. And you had to pass to get "Certified" by B&W, as qualified to take the NRC license exam under the "cold" license rules. This process applied to all original Operators on all 9 B&W plants (at the time), so we're talking many dozens of Operators were trained by B&W in this manner, including even after the TMI accident.
During your certification exam if B&W gave you a SBLOCA, and anytime during the event you let High Pressure Injection run the Pressurizer level off scale high, you flunked, end of story. B&W wouldn't certify you, meaning you couldn't proceed on to take the NRC license exam without some form of remedial training. It’s a fact... my original assistant shift supervisor flunked his certification exam for that exact reason; Toledo Edison dropped him from the Operator training program. There were other casualty events that could also actuate High Pressure Injection, like a large Main Steam line break, or even a simple failure of the ECCS electronic actuation system (false actuation). The point was, if you let High Pressure Injection fill the Pressurizer full, you flunk. It was stressed in training, and not allowed by the operating procedures.
There actually are some valid technical bases for not letting the High Pressure Injection system fill the pressurizer “solid” with water. “Solid”, from an Operator’s point of view, is complicated by the fact that the range of the Pressurizer level instrument does not cover the total span of the tank; rather it stops well before the tank top and bottom. So once your indication is “off scale” you are blind to the actual system water level. Thus it is extremely desirable operationally to keep your level on scale if at all possible. The first important technical reason is that the discharge pressure capacity of the High Pressure Injection pumps (for all plants except Davis Besse) is greater than the design pressure of the Reactor Coolant System such that if they fill the system completely full (losing the steam bubble in the pressurizer) the (simulator) High Pressure Injection pumps can cause system pressure to rapidly increase to the system (code) Safety Valve set point at 2500psi (they are like the “pop” valve on a hot water heater), causing it to lift and blow reactor coolant into the Containment. Also these safety valves weren’t designed to blow water, normally steam, thus the pipe stresses would be severe. If the valve or pipe failed when discharging water slugs it would be a 6” leak in the system.
The second technical reason is a little more convoluted, however it is taught to Operators in training, thus they are well aware of it. It is called Pressurized Thermal Shock (PTS). It is easy to understand in principle. Basically the reactor pressure vessel is solid metal about 8” thick, and at normal full power conditions its average inside wall temperature is the same as the average coolant temperature (they are in contact) or 582F. The outside wall temp is probably slightly lower (depending on the efficiency of the outside thermal insulation), thus a small temp gradient across the 8” vessel exists. But for a thumb rule look, it’s close to equilibrium so just the thermal stress across the vessel from a temperature gradient is about zero. The pressure stress (from 2155PSI) is considerable in the tensile direction, trying to push the vessel outward (pull it apart). The theory of PTS says if the High Pressure Injection is injecting lots of cold water into a very low flow RCS (pumps off), and the system pressure remains high, the cold water starts to cool the inside of the thick wall of the pressure vessel, causing the metal to want to contract. Of course it can’t, because the outside wall is still hot. But that puts a tensile (pulling) stress on the inside wall at that location, which adds to the tensile stress due to just the inside pressure. So I guess the guys that can read their slide rules to 7 significant places (I’m only good for 3) think it is possible to rupture the reactor pressure vessel. That’s not allowed by the rules. It’s similar to a loss of gravity, so the control rods don’t fall into the core (PWRs only), that’s not allowed by the rules either. But this PTS idea becomes problematic in a LOCA situation because almost by definition you encounter a mutually exclusive condition because, after all, you have a big hole in the system and thus no large internal pressure to add to the tensile stress of the cold High Pressure Injection water. Such is life in a Nuke Plant Control Room.
On-the-Job (OJT) Site Training
For folks working in today's world of INPO Certified Training Programs and mature plants that have maybe been operating forty years, understanding the training of my day will be difficult. I will also add that I don't think DBNPP was an anomaly, it was typical of the time, especially for utilities with a single nuke plant. Some of the best overall training we got was the On-The-Job training that inherently goes with being actually assigned at the plant during the latter half of the construction. Just by the very nature of that process of individual system functional and system turn over testing, operations and emergency operations procedure writing, integrated system testing, Hot Functional Testing (HFT), etc., you really learn the whole plant well.
By my recall it was sometime during the major procedure writing effort, especially the large volume of the required Surveillance Tests (STs), that we Operators realized that a normal 4-shift staffing rotation was not going to hack it. There would just be too large an ST burden on normal shift operation activities for them to be running all the STs too, especially the monthly system functional tests. So with much effort on our part we convinced Toledo Edison to adopt a 5-shift manning rotation. The fifth shift was called the Training and Testing shift. That shift provided one day of "relief shift" coverage for the plant, 2 days of conducting STs and 2 days attending training. This of course required that Toledo Edison put another crew in the "cold license" pipeline. Also by my recall it was about 1975-ish, with the approach of winter, we realized enough systems had been completed that 24-hour shift coverage would be needed just to heat the plant during the upcoming winter. So a crew of Equipment Operators (non-licensed position) started shift work. That level of shift coverage continued until about mid '76 when the entire Operator staff went on normal shift work using the 5-shift rotation.
To keep the training in focus I will state our whole station Training Department, up through the initial "cold" NRC licensing exams for the initial Operations staff, was 2 people; the Training Manager and his secretary. The function of the training department was to hire contract trainers to prep us, in one large group, for the scope of subject matter on the NRC exam, not to learn or operate the whole plant, which included basically everything inside the "Owner Controlled Area." For those who might wonder how this approach could possibly work, I'll ask what was a possible alternative. Much of the plant didn't even exist yet, so where would seasoned individual system instructors even come from? Besides, virtually all of the initial operating staff crews down through the Reactor Operator positions were experienced steam plant operators, either from Toledo Edison's fossil plants or US Navy plants. So we all understood the steam plant cycle and had additionally gone through the class room training on the theory of a steam plant cycle. So again, the training was focused on what was required to pass the NRC License Exam. And that type of class room system level training was focused on the Emergency Core Cooling Systems (ECCS), which we received at a "generic" simulator, by contract instructors, using generic procedure guidelines which did not even match our plant. Besides that, how is anybody in that era going to receive operational experience on a whole batch of ECCS systems which don't even normally operate?
By mid 1976 Toledo Edison had put a small group of 5 additional RO candidates and 1 staff SRO candidate through a second round of "cold license" training consisting of the contractor class room training and the B&W Simulator training. Toledo Edison had also been hiring for the non licensed Operator positions, the Equipment and Auxiliary Operator Positions. Thus by the end of '76 when we went on normal 5-shift rotation we had a full complement of Operators for each shift consisting of 2 SROs, 2 ROs, 2 EOs and 2 or 3 AOs. At this time there was virtually no class room training for the non licensed Operators. They totally learned from the shift they were assigned to by OJT, and these Operators were to be the next batch of license candidates.
In January of 1977 we all took our NRC license exams and with a couple partial retakes, we all passed. In about April of 1977 we had initial fuel loading, followed by initial criticality in July of 1977. At the time of the September 24, 1977 event the plant had not even synchronized the Main Generator to the grid for the first time yet. In fact we were in the Power Ascension Test Program, setting up for the initial 15% power plateau testing leading up to the event. At this time, and for the preceding 2 years, the Operating Shift staffs had worked constant 12 hour days, either a 5-12s rotation or a 7-12s rotation; always double shift coverage. At the time of the event I was 33 years old, had never seen a Reactor Trip in the plant (neither had any of my crew), and was a licensed SRO for 8 months.
I'm adding this type of detail because whenever I discuss the DBNPP precursor event of 1977 or the TMI event of 1979, I get questions about my response, specifically about turning off the HPI pumps in response to a full Pressurizer level indication when the RCS was saturated. In today's world it is something so obviously wrong it is something that is hard to believe occurred. And that is true, except in the historical context. The whole PWR industry had the event wrong, because it was never analyzed. Further the Simulators of that era modeled the event wrong and the procedures could not handle it. So I feel it is appropriate for me to ask you "Exactly where could I have gotten the correct information?" All I had available to me was my training and procedures; and I had not been on a Simulator for 2 - 3 years at the time of the event.
The real question to ask is why the TMI Accident is still "officially" being blamed on Operator Error?