I noted in a different blog post some concerns I had where my intrusion detection had a gap caused by log aggregation. Specifically, my Docker web containers forward their logs to a Splunk server for collection and possible analysis later, which means the Fail2ban monitor on that server has no web logs to parse to find potential abuse. While the pieces of the system may vary, it's the bypassing of the logs that is my IDS gap. On simpler server, the logs emitted by a webs server are seen by Fail2ban, and when matches are found, future connections to the server will be blocked for a period of time. This can be done by injecting rules into the firewall, or leveraging IP sets for in-memory lists of IPs to block. Not being able to see the logs is quite a damper on the detection schemes.

I spent a little time researching how to make the other server, which also hosts Splunk so its logs are consumed instead of being forwarded, share its findings with the other. In my search I found CrowdSec (mangled from "crowd security"). As the name suggests, a big feature of this intrusion protection system is distributing "agents" and "bouncers." This met my need of getting the server that was "in the dark" information from the server that's "in the know."

On one or more servers, an agent is installed that can ingest log files, parse them through sets of rules, and determine intrusion attempts and other bad actions. Multiple agents can be configured together, to allow multiple servers to share their information with each other. This information is also shared with the crowd, so IPs that are performing bad actions can be blocked throughout a network, enterprise, and the entire cloud of CrowdSec participants. By running an agent that participates in the crowd, it also receives details of attempts on other systems. The participation in the crowd can be turned off, but is on by default, making it easier to link agents. There are good tutorials if multiple agents are desired, and they should work together in a local cloud, or crowd...I've started using them interchangeably.

Then one or more bouncers are added to the egress or other servers where the firewall protection is desired. These bouncers, like Fail2ban, leverage firewall rules or IP sets to stop the bad actors from even connecting to the server. Unlike Fail2ban, these bouncers are connected to an agent for their banned IP lists. As an agent learns of bad actors, the bouncers are informed and those IPs are banned from the system for some time. This is obfuscated, and all that is logged is that some number of bans has been added or removed. When an agent detects a match, that IP and match is logged, but when those matches come from the crowd, that's when it's obfuscated. The bouncers never do the detection, so it's always obfuscated.

I've used Fail2ban for a long time. Frankly, it's still installed as I test out CrowdSec. Since getting things going, I've seen them both find some of the same actions, and each find actions the other isn't configured to identify. My firewall isn't currently logging when an IP set match is found, but when either reports a ban, traffic from those IPs stops appearing in the log files. It's a little more mysterious if a crowd update has an impact, though, as it doesn't seem to be the case that an update will happen as an "attack" occurs.

I'm tinkering with some firewall logging related to the IP set rules, and hope to see some real value from that missing metric. I've had firewall logging turned on in the past, and it generates a ton of noise. Some of the tips I'm finding about IP set logging seem to be able to curb crazy logs, but while the techniques will reduce repeated entries, it's unclear if it counts the attempts. I'd like to see just one entry per IP in a given time span, like "1.2.3.4 blocked by crowdsec-ipset 200 times in the last 60 seconds." I expect that number to be a lot higher, given how many packets hit a system during a transaction, but I also expect IP set rules to block very early in the connection. Still, packets will come and be blocked...so high numbers seem likely.

I'm also tinkering with the CrowdSec "scenarios" to find those I might use, or add my few tweaks that I liked in my Fail2ban configuration. Both are watching for bad SSH attempts, but Fail2ban is configured to immediately ban any attempts by "root" or three bad password attempts. I also have a Fail2ban rule for finding repeat offenders, so that IPs that are banned too many times in a short number of days get banned for a much longer period than normal; the duration of a ban depends on whether it's a web, e-mail, SSH, or other kind of attempt. For what I serve, and the access I need, I could probably permanently ban any bad actor and not ever impact a real person.

Out of the box, though, CrowdSec has already provided extra coverage. The container server is now blocking IPs that were previously attempting to do bad things. While I haven't got the firewall logs showing the IP set matches, I do see practically zero Log4j exploit attempts on the server. That gave a neat burst of activity on my servers, which has dropped to the normal trickle of hits I expect. I had a Fail2ban rule on the day that the exploit was noticed, so one server was protected (and no vulnerable apps are running, but still the attempts are made). The other sever, though, kept getting tries. Now, not so many. Only 3 in the last 24 hours, versus the hundreds a day prior to implementing CrowdSec.

Another fun use of CrowdSec is that there are metrics collected that can show where the different rules are stopping attempts. https://crowdsec.net/log4j-tracker/ shows "live" data gathered from the crowd to stop the Log4j attempts.

I don't know if I'd put CrowdSec in front of an enterprise system just yet. More on my comfort with it than any flaws it has. It seems like it'd be possible for a bad agent to inject bans into the crowd data, and since it's obfuscated and privatized, this could lead to blocking good actors. This might not sound bad if suddenly some people couldn't get to services being protected, but it could be trouble if it banned if enough people or systems in a mesh. I'm sure this fear could be abated with some more time and study. It's probably also shadowed by what things are being done by companies the size of the one where I work, with its billions of web hits a day.

I do think CrowdSec is easy enough to configure, and stops enough of the right stuff for personal use and smaller enterprises that perhaps don't have the security corps or chops to do bigger. Add an agent, really whether it analyzes your logs or not, then put bouncers on all of the systems that can be accessed from the internet.

I also wonder if I could run CrowdSec in my kinda beefy router. It's already running an embedded Linux, and has IP sets and iptables both. If it could install a firewall bouncer and work with the agent installed on my network, then it could stop all traffic at the proverbial Internet gate. I'm not going to go that far today. So far, the intrusion efforts built into the router do a good enough (it seems) job of protecting itself and stopping a variety of unwanted traffic. It's really those forwarded-to or addressable IPs that I have that I need to protect this way. Today, any system that my router forwards traffic to has sturdy firewall rules, thousands of blocked IPs based on ipdeny.com, intrusion detection with Fail2ban, and now CrowdSec bouncers.