Server Reboot Killed Websites
More accurately, Docker failed to restart any of its containers after the server rebooted due to kernel updates.
Today there were a couple updates applied that required the server to restart. I was idle in a video meeting so I set it on its path. After the server disconnected my SSH, I started a ping every few seconds. After a few minutes, the ping resumed, so I knew the server was back. I usually hit a website or two, but I use Uptime Robot which had sent me a notice when the server stopped responding, and surely would have sent another when the web containers started. Distraction continued, and other meetings took my attention.
A short time ago, I noticed an alert from my CDN telling me that my server had been unresponsive for more than allowed. That message was a couple hours old! Yikes! I checked, and sure enough, Uptime Robot hadn't turned green, and hitting one of the server's websites returned an error message from the CDN.
I just now did good old service docker restart
on the server, and in came the good alerts. The alerts are awesome, but one does need to follow through with them, especially when in a failed state, even if that state is expected for a brief period. I'm unsure what caused Docker to not start the containers. Docker itself was running, but none of the containers are. I'll check some logs, but did need to make it work first. Had it failed when explicitly restarted, I would have looked sooner.
Note, yes, I could load-balance and run these web and application servers redundantly. I do know how, and do have the equipment for HA, but it'd just be practice, given the volume of non-robot traffic on any of the sites.