As I was out dropping the kids at school, the wife sent a text saying the power was out in our home. This is a pretty rare occasion, but something happens every  now and again.

As I returned, i could see the semaphore at the corner of our block was out (and people on the busier street were just blowing through it, not stopping as you're supposed to), so it wasn't just our house. She confirmed and had been in the alley chatting with other neighbors about the same.

We had carpet installers over at the same time to replace the carpet in our family room. They had successfully removed the old carpet and pad, and were in the middle of cleaning the debris when the lights went out. They had to postpone the rest of the cleaning and installation until they could get power, because that's how vacuum cleaners work.

I went down into the dark basement to shutdown the servers as their UPSs blared, and the countdowns dwindled to just one minute remaining in one UPS, 17m in another, and 130m on the UPS for the network gear (routers and switches don't take much). I need to rethink my scheme and either invest in more UPS, or allow the servers to each run with just one power supply backed by battery. I have 3 1500VA supplies, which gives about 30 minutes as configured, but should be able to stretch to more if only one server PSU draws from each. The big server has 4 900W PSUs, and the little has a pair of 200W PSUs, and each will function with only one PSU powered (although both require all PSUs to have power to start). I'll see what I can work out as I continue my slow cleaning of the basement and server spaces. Maybe I'll put one each of the big server's PSUs on battery on a couple of the UPS, and one of the little's on the other with the routers, and then surge-protect the other PSUs. Something like that.

The power flicked back on about an hour later. I got the servers restarted, re-established the firewalls (I really need to spend some time making that automatic), and restarting a couple of Docker containers that didn't start right (I need to Kubernetes that stuff, too).

Work resumed, and no power wavers returned. I checked later, and all of the UPSs had full batteries.

The carpet installers returned an hour or so ago, and seem nearly done with the installation. Once they're gone, we'll return the furniture to the family room, and all will be normal again.

Everyone's healthy.