I got a wave of frustration with my web access logs, so I did a few things over a few days to make them better.

The most pressing was that my access logs were reporting the IP address of the instance of my CDN, Cloudflare, that was accessing my server, not the client at the other end. Cloudflare does offer an HTML header with the real IP of the client, but my web server egress configuration didn't do anything with it.

I use nginx-proxy in Docker as my main egress. With some environment variables set correctly, starting other Docker containers will trigger a reverse-proxy entry in nginx-proxy, allowing for dynamic back-end servers. I learned about the real IP module in nginx. After a little poking, I found the necessary "is it there?" check for seeing that nginx-proxy has the real IP module built in! I added a config file to my server's /etc/nginx/conf.d file that lists all of the CDN networks and sets the modle configuration, and as soon as nginx-proxy restarted, the logs changed and started showing the real IPs of the clients. I tested some things and saw my known IP flying by!

I then thought that my intrusion detection systems, things like fail2ban and Crowdsec. They're working on the few directories filled by the native web server (there's one Apache with some domains and sites that are not yet containerized) and system logs, like e-mail and networking. The Docker containers and some other servers or devices use rsyslog or syslog-ng to send their logs to one syslog-ng server. These are also slurping from the folder that server writes the syslog-ng logs.

After a quick double-check, I saw that while Crowdsec was occasionally identifying the CDN IPs as doing something untoward, even if they were banned, that ban check was after a whitelist check to allow the CDN IPs to the HTTP ports on the server. Looking more, there wasn't an Crowdsec configuration looking at the syslog-ng logs, just that other Apache (and SSH and mail...) logs. There is a way to get Crowdsec to take the syslog-ng input directly. It can start a UDP or TCP collector. I poked around and put some configuration stuff in the /etc/crowdsec/aquis.yaml file to set up the listener. There were a few passes to get the indentation correct, but essentially adding this to the end of the file:

source: syslog
listen_addr: 10.11.12.13
listen_port: 5114
protocol: udp
labels:
  type: syslog
---

I then added a bit to the syslog-ng configuration to forward copies of log messages it receives to that other syslog-ng server /etc/syslog-ng/syslog-ng.conf file. I might be able to move it to its own config, but this was faster.

destination d_crowdsec {
  udp( "10.11.12.13" port( 5114 ));
};

log {
  source(udp);
  destination(d_udp);
  destination(d_crowdsec);
  source(wan);
};

Really the two files just need to match the listening address and port with the forward address and port. Shortly after getting both running, I saw a syslog-ng web log from one of the containers followed by a Crowdsec ban message with the right IP! The ingestion works!

I don't see a way to get fail2ban to take the syslog-ng directly, and it's also not configured to look in the syslog-ng directory. I'm not terribly worried about it, and after I go it working, I realized why. It struck me after I got everything flowing to the Crowdsec acquisition, I remembered the CDN was connecting, not the client. The client isn't connecting, so the neither the Crowdsec bouncer or fail2ban ipsets will never be tested. Further, the only holes in my router for the HTTP ports are the Cloudflare IPs, so there's no way the banned IP could connect.

Still, I got it to work.

Also on the server is my log analyzer, Splunk. I should probably get off Splunk, especially since I'm currently disabled on my instance because I had 11 days of enormous log messages that got ingested; I have to wait two more weeks to do any searching. I can still configure the listener, though. For Splunk, it was a simple creation of a UDP listener, picking the syslog type, and let it go. Then make a similar change to the syslog-ng configuration as above, with the Splunk server IP and port, which is the same server (for the same log ingestion reason), so I used port 5814.

Everything seems to be working. In a couple weeks I can try searching. I should have all the syslog-ng stuff in the logs since my ingestion change. The logs are now tens of megabytes each day instead of the tens of gigabytes during the DDOS period (of my own making). The license violation will clear when there are three or less violations in the last 30 days, which is two weeks away.