Comments SPAM interlock

So another comment spammer took down my server this weekend. It seems that it takes Movable Type over 8 seconds to rebuild the blog.org pages after a comment has been posted (the category pages are large, and get re-written to update the comment count in the summary). If a spammer tries POSTing to several comments pages at the same time, or does so over a relatively short period of time, I get a whole bunch of mt-comments.cgi scripts running simultaneously.

At 8Mb (of working memory) each, it doesn’t take long for them to max out the memory on my wimpy 128Mb box, at which point paging starts, slowing everything down and making the problem worse. As more HTTP requests show up, and cron scripts run, the box starts thrashing (i.e. spending all of its resources moving pages in and out instead of accomplishing useful work). I couldn’t even SSH into the box; the SSH negotiation was timing out after a few minutes.

Usually I have to ask my host to physically reset the server, but this time it was a long weekend. Fortunately I had a remote shell lying around. But it took two days to run su, type my password, and kill off the offending httpd and mt-comments.cgi processes. In the meantime, many other important daemons had been killed due to out-of-memory, and the box was completely ignoring web requests and e-mail sessions; in short, the machine was a mess.

MT-Blacklist is due out today, and I intend to install it, but it won’t help this problem; by the time mt-comments.cgi is being exec()ed, it’s already too late.

So instead I wrote a simple locking wrapper for mt-comments.cgi. It’s in C, so it’s tiny (working memory is 306Kb instead of 8Mb; still way too large, but much better). It grabs a lock file before running mt-comments.cgi, so that only one instance is running at a given time. I’m hoping this will prevent the box from falling off the ‘net the next time a comment spammer shows up.

I’ve also dropped the value of MaxClients in my Apache config, to prevent too many simultanous Apache processes from starting up (since this will also eat the virtual memory system for breakfast).

I wish there was a better way to do load shedding in this context, but I can’t think of one off-hand…

posted at 12:24 pm on Tuesday, October 14, 2003 in Site News | Comments (2)

2 Comments

  1. joy says:

    Sorry to hear about the comment spammer… I was wondering what happened to the box though.

    About MT Blacklist, it appears to work before the mt.cgi is executed in that it checks material in the posting before the posting occurrs. It also halts the posting if there are hyperlinks to dubious sounding Web pages.

  2. Ginger says:

    I was wondering what happened to you. You weren’t here when I checked in Monday.

    I dread the same thing happening to whiterose. You’ll have to keep us posted on how well your spam-combatting measures are working.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.