New Year Cleanup
It's been a couple of days into the new year, and I've been poking at the servers to make sure everything is tip-top.
Well, it's never the case everything is tip-top, but I try.
All of the systems survived the date change without incident, as expected. They also handle daylight savings well, because most of them use UTC anyway. I have been tweaking some system monitoring and alerting, usually leveraging Slack as a warning mechanism, as it's installed on all of my systems, both mobile and desktop. In a few cases I need to continue to leverage e-mail, but that's pretty solid, too, and is tied to my phone and tablet, at the very least.
I've been running through the smaller systems, like this one, shoring up their product versions. This is now running the latest version of the blog software, as are most of the sites or domains running the same. I've been making moves to the OpenJDK as the licensing is getting more confusing and possibly concerning with the Oracle JDKs. So far I haven't had any real issues with those apps.
I continue to migrate things from the long-running server to the new beefy server. I've given my ire over the lack of disk space on the new server (it's got just 400GB of real storage) by leveraging network-attached storage, keeping the e-mail and large database servers off, and centralizing logging to other servers. The axle I keep wrapping myself around is that the next version of firmware (which I can't "find," and Oracle makes hard to acquire--I miss Sun Microsystems) does support the larger drives I've already got installed on the system. I do have two GB ports on both the big server and the big DB and file server (which has TBs of storage), and the big server does have an alluring fiber NAS card (...must keep off eBay...), but for now, really the DB, mail, and logs are the space hogs.
While I'm mentioning axles, I'm also wrapping myself around some CI/CD axles. I'm fortunate to work in a place that has an excellent team of engineers adding all of those features to our development infrastructure. I've toyed with many on my systems, too, but cannot reproduce the convenience they've allowed for me. So I continue to have a slightly manual CD system; I can trigger the CI fine, and have my repos run my tests on independent systems, but I still need to manually bundle and store my containers, and I need to manually trigger the deployment to my container server. Sometimes I jump into the rabbit hole of trying to get one more step closer to automated deployments, but really I just end up wasting what should have been productive authoring time.
While I was checking out my neatly distributed logging, I did notice that my "local" logging stopped working. Of course, my old mechanism was to leverage cronolog to slice my logs into days for both the static analysis tools and the logging server. Since the year rolled over, the "local" consumption of those logs stopped, as the folder had changed. I fixed this quickly enough, and glad to do it on just the third morning of the year; I've done it in the past in February, which leads to baby-stepping through the logs to not break the licensing. I do plan to move away from Splunk (as much as I like it), to ELK, but this change is hard, and I need to either figure out importing the past (again), or dropping it and moving forward.
I'm blathering all of this more for my benefit than any of the random readers who find this. There's a lot of "oh yeah, that's when that happened" and "whoops, I forgot I did that" in here that helps me when I run into trouble. I hope some of it is beneficial to others, at least confirming that their version of weird maintenance isn't totally unique. As well as some of the actually helpful things.
Moving on...