For both protecting my systems and playing with what's out there, I incorporate Crowdsec into my system's intrusion detection and prevention. For other reasons, I updated one of my servers to the latest Ubuntu (not LTS), and an update to Crowdsec balked.

After upgrading, I noticed the server had a seemingly older version of Crowdsec than I expected. It seems that the Ubuntu package uses Crowdsec 1.4, instead of the latest v1.6 from Crowdsec. I was sure I had installed Crowdsec from the source, but maybe in the upgrade it chose the Ubuntu version because it had disabled the third-party sources as part of the installation.

I tried to add the source from the Crowdsec installation docs, but it sadly failed. It turns out that Crowdsec doesn't yet implement Ubuntu's Mantic version (23.10). Probably more correctly, the scripts and package storage used to set it up don't have the ubuntu/mantic versions, although they do have ubuntu/kinetic ready to go.

I had disabled the service and the bouncers on all the servers thinking that it would be pretty quickly sorted. I've checked between, and it still didn't work with the streamlined connections.

Today I gave it a next-level try and discovered that I needed to do was edit the package list to use the kinetic version instead of the mantic version. Fingers crossed that there weren't any OS changes that broke it, and I tried again.

There were some configuration changes that broke it. Insert heavy sigh. Probably due to the couple of version jumps, lingering stuff from the Ubuntu install (maybe breaking the files), or other bits I poked at when I tried fixing it before.

I removed the config and let the install run again, and it behaved better. I edited the config files to include a few of the bits that I added. I use the firewall bouncer, for example, and the Splunk notifier to log events from the services. I also push my logs into dated folders, so I had to add that to the acquisition settings. Also, because I have more than one server that run firewall bouncers that leverage the single instance of the core service, I need to update the configs to use that hostname or IP instead of the ubiquitous 127.0.0.1 that comes out of the box.

A couple edit corrections later, including changing the credentials to the remote bouncers, and all the errors disappeared and the services started again!

I haven't been violated (that I can tell) in the meantime. Fairly, I can't really tell, as there aren't logs from the Crowdsec bouncer that tell when a bounced source is found. Technically it uses the firewall to do the blocking, so it'd be the firewall that isn't telling me (I've tried logging that stuff, but that gets way too chatty...so many bad people trying so many things...).

There are aggressive blocks on my servers to generally prevent direct access. I use a CDN so all the web hosts (like this one) aren't directly addressed on the Internet. I don't strictly lock down the servers to just the CDN hosts, even though they do share what that limit could be, but I do knock out most of the world where bad actors seem to come from. These blocks get the lion's share of the stops, even though Crowdsec's ipset is before it in the list. It's probably the case that more of those IPs aren't doing enough bad things to be in their list; I don't know that they're trying to do a bad thing because I've added entire countries in those blocks. I can guess they aren't entirely legit traffic as the CDN does obscure my host addresses, so there's no "just browsing" way they should reach the servers.