While spending all the time tinkering with the router, some wonky things happened to the other servers.

I use uptimerobot.com to monitor the various web servers I have running on my network. I think it'll monitor more, but I use simple HTTP(S) monitors to ensure that their service on the Internet can connect to my servers on my network. It works pretty well, alerting me within a minute of a failing thing. Since I've been tinkering with my router, a lot of downtime has happened, so a lot of alerts have been flowing.

After I finished reverting my router after this morning's failed attempt to replace it, the monitors almost all returned to normal. The two that continued to fail indicated that the database server wasn't responding to the web servers (or more accurately, the apps they serve). This happens sometimes when too many connections get interrupted, which might have happened with the dozens of times I've vacillated on the router in the last day.

I couldn't connect to the headless server via SSH. Rather, it would connect and authenticate, but it wouldn't login. This has happened when the server gets resource starved, like when the database has too many of these weird connection states. Often I can correct this by logging in via the "terminal," which is through the server's advanced management tool, connecting through a different SSH connection.

I couldn't connect to the advanced management SSH connection. I wandered down to the data center to check, and found that the Ethernet cable had come dislodged from the server's utility port. I pushed it back in and couldn't get it to "click" into place. I remembered that when I put everything on the shelves, It wouldn't click in then, either, and made another mental (and now real) note to replace it when I'm not so pressed for time.

Returning to the desk, I was able to SSH to the service console. From there I was unable to connect to the terminal, and get a login prompt, but while it took my credentials, it wouldn't complete the login process. I exited the terminal and reset the running system. I waited for it to finish booting, and was able to login. From there I was able to restart the database container. Within moments I received an "all good" alert from the monitor for the two services that were failing.

This blog currently uses that database, and I was clearly able to get back in and make new posts.