UPS Dead Battery
I woke this morning to hear a rapid trilling from the network area in the basement.
When I got closer, I could tell it was a UPS screaming for help. On a glance, I could see that one of the UPSs was warning that its battery wasn't working.
A couple quick checks, in as much as you can, didn't reveal anything wrong. While I'd unplugged everything from the UPSs before unplugging them from the wall, I didn't turn them off while I rewired the rack. I did plug the UPSs into the wall outlet before I plugged anything into them, so they all should have only slowly drained for the few minutes they were unplugged to be moved, and had a decent chance to start charging when there was power before there was any load applied.
This is the first time the UPS has been without power since they worked on our house in December. Knowing the power was going out, I took the time to shut down all of the servers and powered off the UPSs. No alerts when they were turned on again. As far as I know, there have been no power interruptions since. A few flickers of lights, as happens, but that's the primary reason for the UPSs--to protect from little brown-outs.
I peeked at the store where I got the UPSs, and they've moved on to the next generation as these are a little old. They don't sell this model any more, nor any replacement batteries for them, and they don't seem to be compatible with the batteries for the next generation. I checked at the battery store nearby, and they don't have them, either. I did find the right APC replacement battery on their site, and then an allegedly compatible one for half that price on Amazon, which is a quarter of the price of a new UPS
The new battery will arrive before next Saturday. Until then, the alarm is silenced, and there's no battery, so everything is taking power just from the wall. Next Saturday, I'll shut down the server that's connected to it (alas, one of the standing servers with only one PSU), move the network switch power to a different UPS (minor network interruption will occur, but nothing should break), quickly swap the battery, and return everything to the right outlets and ports. I just gotta hope it all holds together until then.
I'm also adding UPS monitoring to my list of things to get working. The UPSs have the ability to connect to the servers so that the server can know what's happening. If I can sort out how to connect all three to the one server, I can have it monitor and maybe turn off all the servers when they all notice the power is out. Maybe if I'm really savvy, and the monitors allow, it can know where there's a problem like this, and where there's a single PSU server or which network gear is at risk, and politely turn them off. And, of course, try to notify me with an alert as it happens, too.