Our Liebert UPS started throwing alerts and emailing us every 1-6 minutes.
Events listed in chronological order.
Event: Cleared:Load On Battery
Event: Cleared:Load On Bypass
Event: Active:Load On Battery
Event: Active:Load On Bypass
Event: Active:Inverter DC Voltage Low Shutdown
Event: Cleared:Inverter DC Voltage Low Shutdown
Active Events: 2
The audible alarm on the UPS was going off. I called our UPS contact who we have a support contract with and got them scheduled to come onsite. Unfortunately this happened on a Friday at 3:30PM and they were located 2 hours away. I came back in to meet the tech, who disassembled the UPS, and noticed a slight oil residue on the capacitors and some wires above the caps.
The capacitors weren’t something they keep in stock, there are DC caps, which are black and the size of an oil filter for a vehicle, and taller skinnier AC caps. The AC caps were the ones that had the majority of the oil film on the top of the caps.
Our UPS is currently offline, and switched over to bypass and we are running on straight utility power at this point. Our building has a generator, but still takes 15 seconds to power on and get up to load to cutover, which is 14.9999 seconds to long.
The tech returned on Monday to replace the caps, powered the UPS back on, and the same issue was still present. We talked with the tech and someone at the support company and the next step was to replace some boards in the UPS, of course they try and upsell us a new UPS at the same time. The UPS is around 10 years old, in general with regular preventative maintenance, we should be able to get 30 years out of the UPS.
According to the tech when they replace the board and cutover, it could possible overload the AC circuit and pop the breakers in the UPS, thus shutting down our entire datacenter. This has happened to 1 of 4 customers they had the same issue with other customers with. The whole scenario smells suspicious and the more I am working with them, the less confidence I have in their knowledge.
We called another more local company who works on Liebert, and explained the situation, and of course they said they have never heard of replacing that board causing an overload and popping breakers, so we are waiting to hear back to get a quote for a PM to find out whats actually wrong. We are working with the original company to see if we can rent some rack mounted UPS’s temporarily to move over everything with dual powersupplies, and then we only have to worry about some network equipment that is redundant by having 2 units, but each unit only has 1 power supply.
We ended up bringing in a Liebert tech directly from Vertiv to look at the issue. He turned the UPS on, and within 2 minutes, knew what the issue was exactly. There are 5 hall effect sensors which can go bad overtime, and need to be replaced.
Each sensor was about $80 to be replaced.