At the end of January, there was an inexplicable Denial-Of-Service attack on my server. One of the hosted domains more than the server. An ingeniously simple POST attack (open an HTTP connection, make a POST request, very, very, very slowly dole out bytes) rendered my web server unresponsive. It never crashed, but the pool of workers was occupied enough to stop responding to the few legitimate requests on the server.

I put a quick stop-gap solution in by changing the one page that offered a POST form to a GET form, and then rejecting all POST requests. I then also started a rampage on my firewall by adding all of the IP addresses that participated in the request. My first inspections showed that all of them were coming from known malicious countries like China and Russia, so I blocked all of them. It seemed to be the case that as soon as I blocked the recent offenders, new ones would start trying; I'd block those and more would come--always new ones. A few times I experimented and turned off the firewall blocks and was immediately inundated with a flurry of previously blocked addresses, so I'd turn it back on.

Ultimately, I made a change and forced HTTPS on the server, and that seems to have best mitigated the issue. The connection is attempted, but before any POST traffic can begin, the HTTP server rejects the connection, responding with a redirect to the HTTPS address. Except for the HTTPS, it's all the same, including the POST rejection on the server.

The attacks never try HTTPS, however. From what I can tell, the latency involved with negotiating the HTTPS is the problem, as the attackers don't want their remote systems wasting that time. I'm not entirely sure why that's an issue since the methodolgy of the attack is to consume as much time as possible, so you'd think a few milliseconds on the front end wouldn't matter. Perhaps it's the lack of SSL certificate handling on their attacking bots, but you'd think again that the bots wouldn't really care if the certificates were valid, so they'd just skimp on that. I guess the silver lining is that since they don't bother, the simple solution is working and I don't need to try any harder than I already have.

It's been five weeks since the first attack was noticed. I continued adding the IPs through the month of February, but let them go for the first week in March. Since putting the protections in place, the number of IPs hitting the server has been consistently in the range of 500-1000 each day, and the number of requests consistently near 5000. Of course, except for the few attempts at reverting the firewall.

Today I've reverted the firewall entirely. Now it's back to the protections previously offered, where the world is prohibited from SSH access, and the 15 worst countries (according to security sites) for SPAM are prevented from reaching the SMTP servers. Really, the firewall only opens HTTP(S) and SMTP anyway, except to a few specifically configured addresses; SMTP is filtered from about 76000 IP ranges, representing those higher-risk SPAM countries.

I expect a lot more IPs and network hits, but still I'm hopeful that even if there's a slew of new requests, there won't be such a violent increase in connection abuse. It seems to be that while there are a lot more POST->301 entries in the log files, but there's no significant increase in the server's CPU or network utilization. Really, the worst of it is the network traffic when tailing the log file.

I still shake my head at trying to understand why this happened, and why it's on-going.