I found this quote while reading a news publication: “Sometimes the easiest way to solve a problem is to stop being part of the problem.” That’s because many of the most common problems in pumping systems are of our own making—specifically failure to develop a general scope document (GSD). I will reference the GSD many times throughout this series of articles on root cause failure analysis (RCFA). Why? Becaus I firmly believe (from firsthand experience) the root cause of most pumping system component failures is due to lack of clarity during the development of the project.
Failure Analysis vs. RCFA
Yes, there is a difference. Failure analysis (FA) is replacing the failed component. This is nothing more than parts replacement because the failure mode was never identified. It has been my experience that failure analysis is the typical course of action taken by most facilities.
RCFA, though, involves a comprehensive analysis identifying all of the root causes (physical, human and latent).
To make my point and to set the tone for this series of articles, the following RCFA from 2004 is a classic case for differentiating FA from RCFA, as well as the cost associated with the common strategy of getting the equipment back in service ASAP. This RCFA case study has been condensed from its original 30-page version. Key points and findings have been used to make a solid business case for identifying the root cause of a failure as opposed to component failure/replacement.
Combined-Cycle Power Plant Condensate Pump
Plant Configuration: 400-megawatt (MW) combined-cycle power plant, oil/gas fired, commissioned in 1972, two 200-MW units, load swings daily
Equipment: 700 horsepower (hp) vertical, 1,750 revolutions per minute (rpm) condensate pumps/motors, two 60 percent pumps with one 60 percent installed spare
Situation: The plant began experiencing a number of condensate pump failures on one unit (2B). The site experienced nine failures in the four years, resulting in lost production/downtime. When the plant experienced an equivalent forced outage rate (EFOR) event during peak load, corporate took control and brought in a third-party consultant to resolve the situation. The data below was compiled by the balance of the plant engineering team in conjunction with the third-party consultant. The maintenance staff became quite proficient in replacing the wet end of the subject pump, performing change out in approximately 12 hours upon failure.
History: Over the years, the plan has experienced a high turnover in maintenance personnel. The plant would swap pump wet ends and motors from one unit to another without documenting. Experience level dropped over the years, and the site had difficulty retaining experienced mechanics.
Operations: Same scenario as mechanics, additionally the plant deviated from its operating procedure.
Maintenance/operating cost over 30 years: Upon review of maintenance records, we were able to develop an idea of the life cycle cost (LCC) of the unit 2B condensate pump.
Note: This is valuable information when justifying your solution. obtain this data when evaluating an application.
Rebuild pump: According to plant personnel, until four years ago the pumps would last about four and a half years—therefore, the pump required rebuild seven and a half times over a 30-year period. In the last four years, the pump required an additional eight rebuilds.
Note: Customer expectations/mean time between repair (MTBR) is four years (industry standard is 10 to 15 years)
Cause & Effect
- seal failure
- motor failure
- motor failure
Soleplate out of level:
- pipe stress
- seal failure
Coupling out of tolerance:
- incorrect placement of check valve
- incorrect startup procedure
- combination of above findings as well as incorrect operation of pumps; two pumps in parallel when only one unit is required
Cost to rebuild pump: $60,000 x 7.5 + $60,000 x 8 = $930,000
Labor to remove and install pump:
$5,760 x 15.5 = $89,280
Motor repair: 25,000 x 7.5 + $25,000 x 5 = $312,500
Coupling: $3,000 x 5 = $15,000
Mechanical seal: $5,000 x 7.5 + $5,000 x 9 = $82,500
Total = $1.5 million (approximate)
Lost MW: Best guess based on fleet records is 60 days = 200 MW x 24hrs = 4,800 MW x 60 days = 288,000 MW x $40 (average per MW over 30 years) = $11.5M
Valve damage: No record of repair, however, valve failure did result in EFOR
Main condensate recirculation regulator: Again, no record of repair, failure of main condensate recirculation regulator will result in EFOR
Total cost: $13 million (approximate)
Due to loss of expertise, the pump was not installed or aligned correctly, the motor was not set level, the coupling had a significant run-out and the pump was installed with 0.6-inch pipe stress. The subbase was degraded and crumbling. As a result, the sole plate was out of level 0.2 inches. The motor had a run-out of 0.1-inch shaft to motor face. The coupling was not manufactured correctly with a 0.005-inch run-out on concentricity and, due to the run-out at the motor shaft to face, the coupling face was out more than 0.2 inches. The operators failed to vent the pump prior to startup. In addition, they would start two pumps when only one was needed.
Surprised by the total cost? I call this the cost of doing nothing because the plant only replaced failed components. I have found this course of action to be the norm. For the next 12 months, I will outline a process for performing a structured RCFA to avoid the cost incurred by this case study.