On July 16 last year, a high-speed centrifugal pump failed catastrophically at a large refinery in South America, causing production losses and costing a substantial amount of money for repairs. The inboard bearing of the pump lost lubrication, overheated, and seized up.
Investigation of the pump failure showed that an oil reservoir on the side of the pump contained sufficient lubricant, but it was not getting to the bearing in question. The oil level in the housing was regulated by a pressure differential, but a knob used to adjust the level may have moved, possibly due to vibration, altering the flow and starving the bearing of lubricant. Whether the knob setting was correct or not was never determined, but the failure was clearly due to a lack of lubrication. There was visual evidence of extreme overheating in the bearing housing.
Another contributing factor may have been the flow rate at which the pump was operated. Maintenance personnel testified that the pump was normally run "above the pump curve," but that it was not normally subject to cavitation.
While the failure of a bearing and the loss of a pump may not be unusual in a process where hundreds of pumps are working, it could have been prevented in this case because of numerous early warnings. In fact, an automated motor-pump train monitor and advanced vibration analysis system installed on that pump only four months earlier delivered a steady stream of information to refinery personnel for several days indicating the pump was in jeopardy.
Automated Monitoring
A new technology combines continuous vibration monitoring with the diagnostic and communication capabilities of smart, microprocessor-based instrumentation, automatically and continually assessing the condition of AC induction motor-pump machine trains.
A field mounted instrument receives inputs from six different vibration locations on a motor-pump train, a tachometer reading for shaft speed, motor flux input from a flux coil, and temperatures measured at the motor surface and an ambient location. Each vibration channel delivers 6,400 data points during a collection period of 0.2 seconds, followed by analysis of the data for 15 seconds. The data collection and analysis procedure is repeated every 25 seconds.
Motor-pump train defects tend to have similar failure patterns across a variety of pump installations, and these patterns are used as the basis for automated analysis.
In this process, a machine is continually scanned for the presence of common malfunctions, such as bearing or cavitation problems. The process is streamlined to focus on the measurement location as well as the increase (or decrease) in vibration levels and the rate at which the levels are changing. The focus is on fewer pieces of data, but the parameters are continually scanned for changes, enabling the optimized methodology that can deliver results in a more timely fashion.
This type of analysis, examining less data more frequently against known patterns, provides speed advantages over analysis of the data typically obtained with periodic readings. It lends itself well to an embedded intelligence system capable of recognizing and analyzing common motor-pump problems.
Probable cause and effect relationships based on these patterns help determine whether something is happening internally to justify a warning. Results of current performance characteristics of the machine are transmitted directly through the process automation system using a fieldbus protocol.
An overall machinery health value, in addition to pump health and motor health based on various physical characteristics, can be assigned a numerical rank from 1 to 100, with the top number indicating perfect health. By checking periodically on machinery health values, operators can get a quick fix on the condition of any monitored motor-pump train.
What Happened
A machinery health transmitter was installed on this pump in early March 2006 (see Figure 1). Personnel from the U.S. supervised the installation, and feedback began flowing immediately. As a test, the pump was purposely put into a state of cavitation, which the monitor automatically recognized and reported.
Figure 1. Transmitter installation at the refinery. Note the leads from sensors running back to the control box where microprocessor-based instrumentation gathers vibration and other data every 25 seconds for analysis and transmission via the refinery's automation system.
Early in July, the machinery health transmitter began generating a series of alerts that went unobserved by plant operators. An analysis of the automation system's event-journal revealed "ADVISORY" level alerts on the log as early as July 7, continuing until the time of the failure. The category of the initial alerts was ADVISORY, but several times it changed to the more severe category of "MAINTENANCE," which was a signal for operators to notify the maintenance department of a potential problem.
At 04:22 AM on July 12, a secondary description of "Pump Bearing Problem" appeared for the first time, and that was repeated on later lines of the log. At 12:04 PM on July 12, the level was changed to a primary category description of FAILED. Still, the pump did not actually stop running until July 16, so there was time to act.
The failure mode lasted a few hours, during which emergency action could have been taken to avoid the shutdown. The pump health value trend (shown in Figure 2) deteriorates rapidly from about 60 to 0 over the final 10 minutes. Once the pump was in that state, catastrophic failure occurred very quickly.
Figure 2. Final moments before the pump failed.
The investigation concluded that "the transmitter did, in fact, detect an equipment degradation associated with the pump and did annunciate an increasingly urgent situation." This analytical result and subsequent notification to operators would typically allow a plant to avoid an unexpected equipment shutdown.
Prevention
The automated monitoring system worked well enough to prevent this pump failure. However, there was a breakdown in communications once the automatically generated alerts reached the control room. For one thing, many other devices in the automation system were also generating alerts, and plant operators had their hands full trying to evaluate and respond.
A thermistor installed on the pump bearing housing would have given an indication of the temperature being generated there so that more forceful warnings could have been communicated to the control room.
Finally, a means of prioritizing the alerts coming from the field will better identify those critical to the continued operation of the process so they can be addressed first.
If the pump health values generated by automated monitoring of critically important pumping machines are taken seriously by well-trained operators, failures such as the one at this South American refinery can be prevented.
Pumps and Systems, February 2007