No, I'm not using the Disk Operating System kind of DOS. Rather it seems there's some kind of Denial Of Service kind of DOS going on, directed at my server.

I was trying to do something else on my server and found it was fairly unresponsive. I turned on the monitor (usually powered off) and saw that my CPU was nailed at 100% and that the network traffic was also unusually high. A next peek at the Apache access log showed a flurry of POST requests for / to one of my hosted domains. Already 130K hits today, almost 200K hits yesterday, 100K hits the day before, and a much more normal 4K hits the day before that. It seems that on 31 January, someone decided to bother my server into non-responsiveness.

I peeked at a random selection of a dozen or so if the IPs, and they're all in China. I'm not sure what I did to someone in China that they'd be against this domain, but whatever. I'm on the brink of just blocking China alltogether from the server. They're already filling my firewall rules with 70K lines to stop them from accessing the SSH and SMTP ports, where there had previously been violent probes to break into the server via some of those exploits (all of which I patch as soon as I can).

It took me a few minutes to get Apache and Tomcat to play nice, but I fixed the problem by cutting off POST at the knees. I took a quick copy of the home page for the site, trimmed out some of the bigger or dynamic fat, and made these quick rewrite rules in the Apache server for the afflicted virtual host:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} POST [NC]
RewriteRule ^/$ /index.html [L]

The first simply ensures that the site has rewrite enabled. The second checks to see if the request is POST. The last forces the response to be the static version of the site and ends the connection. There are probably other rules that might be more efficient, and not cut out all POST traffic, but since the site in question doesn't do POST on its own, this will suffice.

Evidently, using POST as a DOS attack mechanism is kind of old school. An attacker will create a POST request, and then very slowly feed it bytes. This consumes an Apache listener, and if the attack holds the port long enough and feeds enough data to it, will cause the listener or Apache or the server to finally crash.

My server was filling its generous, but still limited worker pool, blocking other requests, but was feeding the bytes to the web app server behind it, instead of collecting them in the Apache worker. This, of course, caused the Tomcat server to consume huge memory, but it also has limits and eventually killed the tedious and fat request.

The requests are still coming by the tens per second, or rather are finally coming at the rate of tens per second, but are immediately handed the (presumably cached) static file. I should shrink the file, as it's 11K huge... Now that I've thought of that, I've made a second, smaller file that essentially says "cut it out," and set the rewrite rule to that. It's now 78 bytes per request. Now my requests per second have gone up, but the Apache workers aren't working so hard, and the outbound network traffic should be a little less noisy.

Added to my list is to tweak that rewrite detection and add it to all of my hosted sites. The alternative is to cut off China entirely, which I'm getting to the point that I'm willing to do. I mean, realistically, I probably don't have any real visitors from China to any of the sites I'm hosting.

http://lwn.net/Articles/418017/