I’ve got anti-spam, so why am I still seeing some spam?

So you’ve got lots of shiny spam-detection software (not eyetests or similar rubbish) installed but are still getting some spam on your email and your website? Why aren’t your spam detectors and preventative measures effective at dealing with it?

Basically, the spam detectors are pretty effective, but it’s a problem of scale. The underlying problem is that there’s so much spam now – something like 73% of email is spam just now (I suspect the web is worse). I expect much of the rest is legitimate robots too, like newsletters, automated billing, or notifications about social network activity.

So, we want to trap the spam, while letting humans and good robots through. We can’t use physical ability tests because there are both human spammers who are paid to spam trickier sites manually, and people like me who fail things like Google’s “human” test because we use technology to overcome our physical limitations: there are now robots that are better than me at voice recognition or passing eyetests!

We try to design websites so that the return on investment for spammers is too low (don’t give untrusted users outgoing links automatically, basically). Even so, when we’re using some popular software like WordPress, our site settings don’t give them a return, but most stupid automatic spammers don’t bother to check and still have a go.

After that, the main things we’re trying are rules of thumb to trap spammers (which is usually enough to filter out 90% or so) and to group sites together in informal co-operative spam-fighting networks like blogspam.net, so that once a spammer is spotted, they should get blocked on lots of sites (which blocks a bit over half of the remaining 10%).

Sadly, the rest gets shown to humans for decision. Real comments are so few and far between now that we really don’t want to risk turning real people away and killing discussions.

We used to go after spammers who got shown to humans, but there are now too many spammers and too many service providers who won’t kick spammers off their services: the spammers pay them and we don’t: all we could do was waste their money in support, so they stopped offering any support to non-customers. Is that a flaw in the co-operative nature of the Internet? Can we overcome it? Wish I knew…

This entry was posted in Cooperatives, Education, Training and Information, Wordpress and Blogs. Bookmark the permalink.

2 Responses to I’ve got anti-spam, so why am I still seeing some spam?

  1. Simple solution to all of this- hashcash. I’ve enabled it on my blog, and install it on other web forms I develop, and it stops automated bot spam 100%; dead in its tracks. If eveyone insisted on it for email, it would do the exact same thing. Proof of work systems are hard to crack.

  2. Hashcash is basically requiring the client to run a script for you. It may be enabled on Aaron’s blog, but it’s not required. That blog says “WordPress Hashcash needs javascript to work, but your browser has javascript disabled. Your comment will be queued in Akismet!”

    Sadly, hashcash is more spam snake oil, with false claims like “blocks all spambots”. It’s worse than useless: it’s wasting the battery of legitimate client devices, which are often mobile and battery powered, but does nothing to test whether the comment is spam. Human spammers will still get through and so will more sophisticated spambots that can run javascript or farm it out to their zombies.

    At the moment, sites using hashcash probably aren’t getting hit by spambots because there are easier targets and javascript execution is not yet common on spambots, but hashcash is not any extra spam protection itself.

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy this password:

* Type or paste password here:

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>