Half-tweaks and Outages
Running through a few updates yesterday, and I managed to knock this blog off the server and kill the SpamAssassin on the other one. As if that wasn't bad enough, the Internet router stopped working for a little bit, too.
Just a quick, normal, periodical update of my systems' PERL modules triggered a few goofy reactions on my systems. At least one of which was a head-slapper.
During the PERL modules updates, the DBI::mysql complained and failed at the tests because my local user didn't have a login without password, so in order to pass the tests, I changed the user@localhost to have no password, and access to only the "test" database. This allowed the tests to pass. What I had forgotten, though, was that this blog used that same user to access a different database; that I found out this morning when I thought to make this mental note. A quick creation of a new database user and change of the blog settings, and all is good again. A quick scan of the other blogs and sites hosted on the server tells me that nothing else has (yet) been negatively affected by the updates.
On the other server, though, there was another quirk as the Mail::SpamAssassin package was updated. It happened that the anti-spam daemon then failed to start. No errors were noticed as the PERL update ran (that is, it didn't stop with any failures; I wasn't watching the output of everything, after all). I noticed this because there were a few hundred extra e-mails waiting in my inbox. I am pretty glad that the last hosted e-mail domain (that isn't mine) had just moved their service to their own internal server before this happened. I don't expect too many people get as much junk as I do (I've had the same e-mail address since acquiring my domain name in the early 1990s), but I imagine that the swell in accepted mail will greatly impact any mailbox on the system.
The SpamAssassin error was caused by a pretty simple oversight, either in the automated installation or through a warning that scrolled past in the flurry of messages, wherein the new software didn't get any updated rules, and, in fact, had no rules and therefore was failing to start. It, of course died, with a cryptic error. I tried rebuilding the package from the source that the CPAN update had downloaded, and again had no errors. My head-slapping moment came when I found out that it failed by running the daemon in verbose, non-daemon mode and watching it scream that it didn't have any rules as it died. This seems like a pretty fundamental failure, and probably warrants a better error message.
A quick run of the rules updater, and the daemon started just fine, happy as software can be, and with nary a "thank you" for correcting the situation...
The server on which this blog is running isn't nearly as fast as the other server (one of these days I'll move these websites to a faster system, I re-promise again), so it's still running the update (also hampered by the CPAN setting of "ask" instead of "always" when a prerequisite is found...the other system blew by these, this one waits for me to say "yes" every time...).Upon its completion, I'll be sure to run the anti-spam update...well, I should be able to do that now; it's passed the SpamAssassin update...
This one server hosts a few of these blogs, its own database and e-mail servers (for supporting the blogs), a Java continuous-integration build-server, and a few thin-client desktops. The faster one hosts a few less websites, database, e-mail, DNS, and another build server. This one is a slightly too-old Sparc 500MHz near-relic, while the other one is at least a close-to 2GHz Intel. A change is overdue, I know.
Unrelated, except that it involves access to the servers, but also curious, was a few-hours long outage at the home -office. I've a small network of hosts on the Internet, and none of them was reachable, indicating to me that a problem with either the Internet router or the external switch was afoot. As I was remote, I made a mental note to reboot these devices upon my return. Before I did return, however, the machines all came back on-line (along with a flurry of e-mail alerts that the servers couldn't see each other), so the router must have been horked somehow and reset itself. Maybe the ISP was doing some maintenance (it's their router) without warning me. A quick peek at the exposed systems showed a simultaneous lack of activity, no signs of system compromise, and up-times indicating that there wasn't a power issue in the server room.