The Blog of Harald

So another comment spammer took down my server this weekend. It seems that it takes Movable Type over 8 seconds to rebuild the blog.org pages after a comment has been posted (the category pages are large, and get re-written to update the comment count in the summary). If a spammer tries POSTing to several comments pages at the same time, or does so over a relatively short period of time, I get a whole bunch of mt-comments.cgi scripts running simultaneously.

At 8Mb (of working memory) each, it doesn’t take long for them to max out the memory on my wimpy 128Mb box, at which point paging starts, slowing everything down and making the problem worse. As more HTTP requests show up, and cron scripts run, the box starts thrashing (i.e. spending all of its resources moving pages in and out instead of accomplishing useful work). I couldn’t even SSH into the box; the SSH negotiation was timing out after a few minutes.

Usually I have to ask my host to physically reset the server, but this time it was a long weekend. Fortunately I had a remote shell lying around. But it took two _days_ to run su, type my password, and kill off the offending httpd and mt-comments.cgi processes. In the meantime, many other important daemons had been killed due to out-of-memory, and the box was completely ignoring web requests and e-mail sessions; in short, the machine was a mess.

MT-Blacklist is due out today, and I intend to install it, but it won’t help this problem; by the time mt-comments.cgi is being exec()ed, it’s already too late.

So instead I wrote a simple locking wrapper for mt-comments.cgi. It’s in C, so it’s tiny (working memory is 306Kb instead of 8Mb; still way too large, but much better). It grabs a lock file before running mt-comments.cgi, so that only one instance is running at a given time. I’m hoping this will prevent the box from falling off the ‘net the _next_ time a comment spammer shows up.

I’ve also dropped the value of MaxClients in my Apache config, to prevent too many simultanous Apache processes from starting up (since this will also eat the virtual memory system for breakfast).

I wish there was a better way to do load shedding in this context, but I can’t think of one off-hand…