The morning started out with some simple updates, one of which wanted the server to reboot.

This happens sometimes, and when I have the time to watch, usually no big deal. The typical problem is that the firewall rules don't get updated, or get updated after Docker starts and messes with the Docker rules; restarting Docker usually fixes that.

Today also had some bad block notifications on /dev/sdb, so I flipped on the "force fsck" flag, and restarted the server.

It came up complaining about bad super-blocks, and went into a safe mode. Couldn't change anything on the disks because everything was mounted read-only, which also prevented most things from starting. I noticed the /forcefsck file was gone, even though it hadn't run, so I tried to reboot it again.

Now it doesn't even get to the read-only mounted running OS.

I'm created an updated server and workstation USB boot set, so I can see if I can correct the disks, or re-install the whole thing.

90% sure I can get it running again from formatted disks today, thanks to Docker and GitLab stowing my stuff, and a lot of config copied to my main desktop. The remaining 10% is reserved for the nuance of some of the co-dependent services, and whether or not my MongoDB (one of the dependent services) back-up is ready to be restored! 100% sure it will eventually work; 99% sure it will eventually be unnoticed by anyone else...except during this period of outage.