Comment spam load issue
Hi everyone, my name is Jay Allen and I am the Product Manager for Movable Type. I'm writing today to address—what else?—comment spam.
This is an issue that, as many of you know, I have spent several thousand waking hours working on since its first appearance back in the fall of 2003, both as the author of MT-Blacklist and as the maintainer of the Comment Spam Clearinghouse. This is an issue which Six Apart takes very seriously, as evidenced not only in the improvements in Movable Type v3.x but also, in some part, by my hiring to this position.
Over the last month, we have been devoting a great deal of resources to solving the comment spam problem once and for all and making it a non-issue, not just for us in the Movable Type/TypePad world, but also for all weblogs regardless of publishing tool. Our preference is towards solutions that scale to the entire weblog medium, not those which merely move the burden from one site to another, from one tool to another, or from spammers to users.
Identifying the Problem
Recently, however, there have been a number of reports about the escalating effect of comment spam on Movable Type installations, especially evident in shared hosting environments. At first, we assumed that these problems were caused mainly on legacy systems (i.e. MT 2.x) running without the benefit of the modern anti-spam measures (e.g. TypeKey, comment moderation, MT-Blacklist v2.x, etc.) built to protect Movable Type installations. After further analysis and load testing, we've actually found that this is not the case.
In fact, we have found that there is a fairly major bug (in terms of effect, but not code size) which causes page rebuilding even in the case of a comment submission which would be moderated and hence should have no effect on the live page. This means that even if you are using comment moderation in Movable Type and even force moderation in MT-Blacklist, your server load is impacted just as if a comment had been posted to the live site. This bug has been fixed in development.
In addition, we have found another less severe instance of unnecessary database connections which would normally be associated with dynamic pages, even if dynamic templates are not in use. This would adversely affect any customer not using static pages by adding the overhead of dynamic files on top of the normal load caused by rebuilding of static files. This has also been fixed in development.
These two bugs are, in high probability, the causes of the extreme server loads that our customers have been experiencing under the load of a severe spam attack.
We are currently testing these fixes both in-house and with a number of web hosts who were among the first affected by the problem. We will have these fixes released to you as soon as the testing is complete. There is no higher priority to us than making sure that our customers and their websites are protected from the effects of these malicious attacks. We expect to give you a firm date for availability of this patch within 48 hours.
What To Do Now
In the meantime, one way you can help protect your system and mitigate the effects of both problems is by enabling dynamic templates. Under normal conditions, there are many factors to consider in choosing dynamic templates vs. static templates. In general, the higher your site's traffic is, the more beneficial static templates are to you. However, since spam attacks are rapid requests that would cause rebuilding in the case of static pages, the sweet spot is moved far towards dynamic templates, even for high traffic sites.
If you would like to change your templates to dynamic, you should check out the Dynamic Publishing section in the Movable Type documentation and also Elise Bauer and Arvind Satyanarayan's tutorial on the subject.
When setting up your dynamic pages, choose the "Build Only Archive Templates Dynamically" option. This choice means your archive templates will not be rebuilt upon comment or TrackBack submission. If you still experience high loads, you can choose the third "custom" option and set all of your templates to dynamic.
This is also a good time to mention our TypeKey authentication service which has proven very effective in stopping weblog spam. If you are interested in setting up TypeKey, check out our public TypeKey tutorial posted today on ProNet.
What's Next
While we realize that these recommendations may not be your normal preference, they should keep your servers responsive despite any severe attacks until we can release the patch. We are sorry for any inconvenience this may have caused and I assure you that we are working tirelessly to remedy the situation.
On a more personal note, I would have preferred the circumstances surrounding my first post here to be somewhat (or completely) different, but there will be time once this issue is solved for me to address the past, present and future of this software as it deserves.
I want to thank you all for your perseverence through these serious problems and for helping us see clearly where the problems lie.
Categories
News65 TrackBacks
Listed below are links to blogs that reference this entry: Comment spam load issue.
TrackBack URL for this entry: http://www.movabletype.com/cgi-bin/mt4/mt-tb-nospam.cgi/4
MT Plus Comment Spam Equals Dead Site we had some downtime today... (categories: moveabletype weblogs) Jay Allen finally comments on recent MT Comment spam load issues I got hit yesterday, and had to close comments to everyone (categories: moveabletyp... Read More
Durch vermehrte Kommentar-SPAM-Angriffe haben eine Reihe von Movabletype-Weblogs mit argen Server-Last-Problemen zu kämpfen. Wie der Produkt-Manager Jay Allen heute auf Movable Type bekannt gab, wurden bei der Untersuchung der Probleme auch zwei ... Read More
The weblog comment spam problem has implications beyond crowded inboxes for users. Even with tools such as the incredible MT-Blacklist (which has blocked or moderated tens of thousands of comment spams on birdhouse-hosted blogs in the past few months),... Read More
Ðàçðàáîò÷èê ïëàãèíà MT-Blacklist âûñêàçûâàåòñÿ íà òåìó íàãðóçêè íà ñåðâåðû îò ñïàìà â êîììåíòàðèÿõ áëîãîâ íà áàçå Movable Type. Ìåíÿ, êñòàòè,... Read More
http://www.movabletype.org/news/2004/12/comment_spam_load_issue.shtml... Read More
MovableType has the distinction of being the premier weblog application, but also has the same distinction of thus being the... Read More
Just bringing this out where even more people will see it. 6A is working on a solution for us. Recently, however, there have been a number of reports about the escalating effect of comment spam on Movable Type installations, especially... Read More
Jay Allen of MTBlacklist fame and now (like many MT devotees) an employee of SixApart has posted that the company has discovered a number of bugs in MT 3 that increases server loads and pisses off web hosts. He reports that a patch is currently being d... Read More
Funny, with all the hoopla about comment spam lately, I haven't gotten a single spam for over a week. My MT-Blacklist log is empty. My anti-spam strategy of boring content must be working. Especially now that my Google Page Rank... Read More
This website is still getting a bit of comment spam, I received seven today. I have banned the worst offenders as best as I am able. Apparently, others have been not so fortunate and are getting thousands per minute, which... Read More
Very late recording and a very tired Podcaster. But as they say in Hollywood the show must go on and I am not to disappoint my dedicated listeners. Shout out goes to Podcasthost.com for putting legal BitTorrent feeds up for... Read More
Very late recording and a very tired Podcaster. But as they say in Hollywood the show must go on and I am not to disappoint my dedicated listeners. Shout out goes to Podcasthost.com for putting legal BitTorrent feeds up for... Read More
Very late recording and a very tired Podcaster. But as they say in Hollywood the show must go on and... Read More
Comments are temporarily disabled until this issue is resolved. grrrr what a pain in the arse. Bloggers using Movabletype take note: MT developers are working on a patch to plug this security hole. Hopefully they'll release it in a few... Read More
Jay Allen finally comments on recent MT Comment spam load issues I got hit yesterday, and had to close comments to everyone (categories: moveabletype weblogs)... Read More
First a warning from my web hoster saying weblog installations across the board are cauing problems with server loading and then this announcement from MovableType... Hmm... Looks like it this time, not only are the customers going back to MT... Read More
I am swimming in spam. Every where I go, every direction I look, every medium I deal with, I am being spammed. Spam in my email box I can handle—my spam filter manages that well enough that I can ignore the problem. It’s all of the other sp... Read More
I've been having a problem with MT-Blacklist 2.x for a while now; it stopped saving my preferences. After loads of fruitless and frustrated (and frustrating) Google and code searches, I finally found an oblique reference to the same problem with... Read More
update mt_entry set entry_allow_comments=2 where entry_allow_comments=1; select count(entry_allow_comments), entry_allow_comments from mt_entry group by entry_allow_comments; +-----------------------------+----------------------+ | count(entry_allow_co... Read More
Commenting is closed until further notice. We'll turn comments back on, as soon as Movable Type addressing the problem with a patch, update, or a new solution. Until then, please use our contact page to discuss BBS 05 with us.... Read More
If you read any other blogs but this one, you've probably heard that MovableType blogs are currently being hammered by... Read More
I’m pleased to announce that Mozilla users can finally use the imagemapped links in the header, above. Thanks to Seline for pointing the problem out in the first place - although I was spurred to action mainly by my own... Read More
On Xanga there were those pesky random proppers. Here, I have to put up with significantly more aggressive vandals. Over the past week or so, I've experienced an influx of comment, trackback and referral spam. The problem was so serious... Read More
TITLE: Netcraft: Comment spamming taking its toll on MT URL: http://blogs.msdn.com/alexbarn/archive/2004/12/18/323960.aspx IP: 66.129.67.202 BLOG NAME: Alex Barnett blog DATE: 12/18/2004 02:32:04 AM TITLE: Netcraft: Comment spamming taking its toll on MT URL: http://blogs.msdn.com/alexbarn/archive/2004/12/18/323960.aspx IP: 66.129.67.202 BLOG NAME: Alex Barnett blog DATE: 12/18/2004 02:32:04 AM Read More
The Movable Type Blog has some very important information not only for those of us who run their product to produce our own blogs, but for researchers interested in blog comments and comment spam as well. < ..snip.. >Over the... Read More
Comment spam load issue Hi everyone, my name is Jay Allen and I am the Product Manager for Movable Type. I'm writing today to address—what else?—comment spam. This is an issue that, as many of you know, I have spent... Read More
Some of you out there that have blogs that are powered by Movable Type will have recently experienced problems relating to high server load caused by a number of bugs in MT and the way it handles the rebuilding of... Read More
Quedan pocos blogs con Movable Type en Planeta Eurielec, pero alguno resiste. Hoy aparece en Slashdot un artículo sobre comentarios de spam en Movable Type, los problemas que acarrea y las posibles soluciones. Parece que incluso cuando hay algún plugin... Read More
So this morning I logged in to my blog and saw that I had over a hundred new comments. Naturally my first assumption was that people just really liked my latest post about kitten rape. Alas that turned out not... Read More
Even with MT-Blacklist, the amount of time I have had to spend dealing with comment spam has been annoying lately, and apparently I am not the only one who's having issues. I am trying to decide... Read More
Everyone who uses Movable Type probably saw this item in the news section of the main admin screen, but I think its worth commenting on. Jay Allen discusses the exploding problem of comment spam, and how comment spam is leading... Read More
In the last few months, comment spam has become a huge issue, recently culminating in many hosts taking their customers' MT blogs offline. In related news, Six Apart has hired Jay Allen of MT-Blacklist fame to help them combat comment... Read More
I've just been notified that my host had to disable Movable Type comments because a sudden comment spam attack of DDOS proportions was bringing down... Read More
Would you be surprised to know that some people who work in the search engine "industry" know who is responsible for a lot of the comment spam out there? I met some of them recently. And some of them even have blogs of their own. Seriously. I haven't w... Read More
Would you be surprised to know that some people who work in the search engine "industry" know who is responsible for a lot of the comment spam out there? I met some of them recently. And some of them even have blogs of their own. Seriously. I haven't w... Read More
During the last twelve months spam has become a serious threat to many bloggers. Some very nasty people have figured out to use automated spam bots to flood especially moveable type blogs with links and spam advertising. The problem... Read More
We've all learned to deal with email spam. We've accepted it, and we've moved on. But lately it seems that I'm being a victim of more than just email spam.... Read More
I originally switched to MT3 because I was having problems with my MT2 installation locking up due to comments spam. Looks like the problems may not be solved with MT3. I guess I've just been lucky so far.... Read More
The comment spam problem has taken a new turn recently: it's become so much of a problem that it's taking down the servers hosting the blogs they're polluting. In response, some hosting providers are disabling comments and even blogs system-wide.... Read More
I just deleted a little over 200 spam comments from here. Yay for filtering and thus being able to select all the offending comments for deletion at once. Looks like there's a problem at the MovableType level and it's not... Read More
Over the past week, I've been sharing the love by converting a couple of blogs for my friends and family. I'm going to take you through the changes you'll make to each template, from the Main Index to the Individual Archive to the Comment Pending page. Read More
It looks like it's not only Zoë and me having problems with masses of comment spam in Movable Type. Slashdot posted a story on Saturday saying that it's becoming a huge issue, largely due to the serious machine overload a... Read More
I was the victim of a significant blogspamming attack about a week ago, just after I upgraded the version of Moveable Type this blog runs on to the latest version, 3.121 (see related stories at Slashdot, Moveable Type, and Netcraft).... Read More
[Sigh] Blogging? It's been really hard for me to do for the past couple of weeks. The election kind of left me feeling down and out and for a lot of reasons coming to vent about it on my website just seemed so god-damned trite. Not that there have'nt b... Read More
I posted earlier about revamping the design here at Waveflux, and now plan to get on it sooner rather than later. The idea appeals to me more and more; I've scouted about for design ideas (for inspiration, not for theft)... Read More
I see that MT has hired Jay Allen, the original comment anti-spam king as a product manager. Good job! His latest post about the subject, though Movable Type Publishing Platform: Comment spam load issue is totally lacking in practicality. He... Read More
Chasmyn mentioned Six Apart's failure to resolve the server-crashing comment spam problem last week and now Jay Allen's addressed the issue on the official MT site.
Read MoreMovable Type 3.14 �布了 Read More
SixApart reacts to the recent comment spam issues that drained bloghosters’ servers by putting unnecessary on the systems. They officially released Version 3.14 to the public. This release mainly comes with the following changes: Unnecessary reb... Read More
I have blogged about comment spam until I'm blogged out. The spam hasn't stopped, but several different kinds of fixes have eased the problem. The one I'm not happy having to do is close off comments in the archives. For some odd reason, I feel the 'sp... Read More
Today I tried to migrate my weblog from Movable Type into WordPress. I like the WordPress interface a little bit... Read More
For the moment I seem to have won the fight against comment spam.I'm using a combination of apache dirty tricks and MT-Perl hackery to refuse access and posting to identifiable spambots. Read More
The latest version of Movable Type has been released. Version 3.14 fixes the issue of extreme loads witnessed on servers under the strain of a massive spam attack. Because these attacks are increasing in both frequency and severity, it... Read More
via eweek.com Movable Type Fixing Bug as Spam Clogs Blogs bloggerµéÀº ÇѹøÂëÀº comment spam·Î ¾Ö¸¦ ¸ÔÀº ÀûÀÌ ÀÖÀ»°Ì´Ï´Ù. ƯÈ÷³ª movable typeÀ» »ç¿ëÇÏ´Â bloggerÀÇ °æ¿ì¿¡´Â ´õÇÏÁÒ. ^^ ÃÖ±Ù movable type¿¡¼ÀÇ comment spam ¹®Á¦°¡ À̽´°¡ µÇ¾ú¾ú´Âµ¥.. À̸¦ Ç... Read More
This is an example of how to respond to an issue. Months ago I made the decision to stick with Six Apart's MovableType despite the bruhaha over the pricing. I figured the dust would settle, Six Apart would figure things... Read More
X-mas-Card-thumb.jpg">[Sigh] Blogging? It's been really hard for me to do for the past couple of weeks. The election kind of left me feeling down and out and for a lot of reasons coming to vent about it on my website just seemed so god-damned trite. No... Read More
[Sigh] Blogging? It's been really hard for me to do for the past couple of weeks. The election kind of left me feeling down and out and for a lot of reasons coming to vent about it on my website just seemed so god-damned trite. Not that there haven't b... Read More
I've just released MT-Blacklist version 2.03-beta for immediate public testing. This release contains the following changes from v2.02: Now using CommentFilter API callback to force moderate (fixes issue discussed here) Added re-initialization function... Read More
Well, I did the right thing today -- twice. Damn right I'm bragging. First off, it turns out that the FreeBSD Foundation has run into a (good!) problem: its donations have been too big. In order to keep its US charitable status, it needs to h... Read More
Due to the comment spam I've been getting over the last week, I've updated this blog (which runs on Movabletype) to the latest version AND enabled Typekey logins. Typekey is a free service and registering with them enable you to... Read More
