Root Cause Failure Analysis Case Study

I found this quote while reading a news publication: “Sometimes the easiest way to solve a problem is to stop being part of the problem.” That’s because many of the most common problems in pumping systems are of our own making—specifically failure to develop a general scope document (GSD). I will reference the GSD many times throughout this series of articles on root cause failure analysis (RCFA). Why? Becaus I firmly believe (from firsthand experience) the root cause of most pumping system component failures is due to lack of clarity during the development of the project.

Failure Analysis vs. RCFA

Yes, there is a difference. Failure analysis (FA) is replacing the failed component. This is nothing more than parts replacement because the failure mode was never identified. It has been my experience that failure analysis is the typical course of action taken by most facilities.

RCFA, though, involves a comprehensive analysis identifying all of the root causes (physical, human and latent).

To make my point and to set the tone for this series of articles, the following RCFA from 2004 is a classic case for differentiating FA from RCFA, as well as the cost associated with the common strategy of getting the equipment back in service ASAP. This RCFA case study has been condensed from its original 30-page version. Key points and findings have been used to make a solid business case for identifying the root cause of a failure as opposed to component failure/replacement.

Combined-Cycle Power Plant Condensate Pump

Plant Configuration: 400-megawatt (MW) combined-cycle power plant, oil/gas fired, commissioned in 1972, two 200-MW units, load swings daily

Equipment: 700 horsepower (hp) vertical, 1,750 revolutions per minute (rpm) condensate pumps/motors, two 60 percent pumps with one 60 percent installed spare

Situation: The plant began experiencing a number of condensate pump failures on one unit (2B). The site experienced nine failures in the four years, resulting in lost production/downtime. When the plant experienced an equivalent forced outage rate (EFOR) event during peak load, corporate took control and brought in a third-party consultant to resolve the situation. The data below was compiled by the balance of the plant engineering team in conjunction with the third-party consultant. The maintenance staff became quite proficient in replacing the wet end of the subject pump, performing change out in approximately 12 hours upon failure.

History: Over the years, the plan has experienced a high turnover in maintenance personnel. The plant would swap pump wet ends and motors from one unit to another without documenting. Experience level dropped over the years, and the site had difficulty retaining experienced mechanics.

Operations: Same scenario as mechanics, additionally the plant deviated from its operating procedure.

Maintenance/operating cost over 30 years: Upon review of maintenance records, we were able to develop an idea of the life cycle cost (LCC) of the unit 2B condensate pump.

Note: This is valuable information when justifying your solution. obtain this data when evaluating an application.

Rebuild pump: According to plant personnel, until four years ago the pumps would last about four and a half years—therefore, the pump required rebuild seven and a half times over a 30-year period. In the last four years, the pump required an additional eight rebuilds.

Note: Customer expectations/mean time between repair (MTBR) is four years (industry standard is 10 to 15 years)

Cost to rebuild pump: $60,000 x 7.5 + $60,000 x 8 = $930,000

Labor to remove and install pump:

$5,760 x 15.5 = $89,280

Motor repair: 25,000 x 7.5 + $25,000 x 5 = $312,500

Coupling:  $3,000 x 5 = $15,000

Mechanical seal: $5,000 x 7.5 + $5,000 x 9 = $82,500

Total = $1.5 million (approximate)

 

Lost MW: Best guess based on fleet records is 60 days = 200 MW x 24hrs = 4,800 MW x 60 days = 288,000 MW x $40 (average per MW over 30 years) = $11.5M

Valve damage: No record of repair, however, valve failure did result in EFOR

Main condensate recirculation regulator: Again, no record of repair, failure of main condensate recirculation regulator will result in EFOR

Total cost: $13 million (approximate)

Findings

Due to loss of expertise, the pump was not installed or aligned correctly, the motor was not set level, the coupling had a significant run-out and the pump was installed with 0.6-inch pipe stress. The subbase was degraded and crumbling. As a result, the sole plate was out of level 0.2 inches. The motor had a run-out of 0.1-inch shaft to motor face. The coupling was not manufactured correctly with a 0.005-inch run-out on concentricity and, due to the run-out at the motor shaft to face, the coupling face was out more than 0.2 inches. The operators failed to vent the pump prior to startup. In addition, they would start two pumps when only one was needed.

Surprised by the total cost? I call this the cost of doing nothing because the plant only replaced failed components. I have found this course of action to be the norm. For the next 12 months, I will outline a process for performing a structured RCFA to avoid the cost incurred by this case study.

Read more Pump System Standards by William Livoti here.