I have a new installation of Splunk running, and connected the big server to the new Internet connection.

Splunk has been a bear, but I think I have it back in its cage.

I can't undo whatever tweak I put into my original Splunk installation to get it to recover. I've exploded a new instance in a different folder, edited its config to have a few things like the host name and SSL certs, and got it working. It ingested the log folders (with many licensing warnings because it slurped so much at once), and things started appearing in the search. I recreated the HTTP event collector, and set about configuring the HTTP event emitters to use the new collector. It all works. I need to recreate some of my saved searches, and maybe tweak some of the things into separate indexes, but it "works" again.

While tinkering with it, I considered abandoning it for some other analysis alternative. ELK jumps out as an easy and free option. I do poke at the logs on occasion, but more and more it's just to have a complete system than it is to have a useful analysis tool. I've looked at a few, but especially since I'm not relying on it for anything critical, I don't want to offload it or pay anything.

Since I was poking at my computers, and in response to a few warnings from my service monitors that the DSL internet was interrupting, I spent a little time to get the big server to use the new fiber service.

I had already connected an Ethernet cable to the router, but hadn't plugged it into one of the ports on the server, so I went and did that. Job half done.

I poked on the server to try to tell which of its ports was now connected but not addressed, but I couldn't tell. So I figured I'd wing it. I configured two of the ports with WAN IPv6 addresses in the netplan file, and "turned them on." One of them successfully connected, so now I knew which it was. I removed the other configuration, completed the setting for the right one, and turned it on again. It works! Addressing job done, now the rest of it.

For all intents and Internet purposes, the server only serves web sites. It does more in its Docker containers, but none of the other things are directly available to the Internet. I tinkered with DNS to point the appropriate proxied entries to the new host, and very quickly received a bevy of new "service down" warnings. I tweaked the routing rules on the server so the new port was the default gateway, and all the alerts cleared.

Prior to netplan I was able to multihome a machine and have it respond to traffic into any of its interfaces by responding through the same interface, which is how I expect this to work. I can't get it to behave in netplan, but setting the default gateway does the trick. In the end, the server will have only the one external port, and won't use the LAN for Internet access, so no worries.

This site is hosted on the other web server, so this isn't evidence it works, but you can hit https://campiakewe.org/ (one of the other sites it does host) and see that it comes up just fine. I have a CDN configured in front of it, so it's really about them refreshing their internal details. In all my checks, all the sites are going through the CDN and coming in through the right port, and all of my monitoring is reporting healthy.

Eventually I'll go disconnect the other router from the big server. I'll also disconnect it from the main WiFi router, which is using the DSL as a fallback for when the fiber connection flakes. Then I can unplug the DSL and cancel its service. I still have to cancel the other fiber service, although I did pause it, so I'm not being billed. I have a small pile of equipment to return, and then I should be done with them at home.

The same company is still the DSL provider at our house in Wisconsin, because there are no other options. They only deliver 1.5Mb/s service out there, but it's enough for the cameras and little bit we use the Internet when we're there, although the kids now know what buffering is. I'll upgrade it to something faster if it becomes available, but for now, it is what it is.