I’ve got anti-spam, so why am I still seeing some spam?

So you’ve got lots of shiny spam-detection software (not eyetests or similar rubbish) installed but are still getting some spam on your email and your website? Why aren’t your spam detectors and preventative measures effective at dealing with it?

Basically, the spam detectors are pretty effective, but it’s a problem of scale. The underlying problem is that there’s so much spam now – something like 73% of email is spam just now (I suspect the web is worse). I expect much of the rest is legitimate robots too, like newsletters, automated billing, or notifications about social network activity.

So, we want to trap the spam, while letting humans and good robots through. We can’t use physical ability tests because there are both human spammers who are paid to spam trickier sites manually, and people like me who fail things like Google’s “human” test because we use technology to overcome our physical limitations: there are now robots that are better than me at voice recognition or passing eyetests!

We try to design websites so that the return on investment for spammers is too low (don’t give untrusted users outgoing links automatically, basically). Even so, when we’re using some popular software like WordPress, our site settings don’t give them a return, but most stupid automatic spammers don’t bother to check and still have a go.

After that, the main things we’re trying are rules of thumb to trap spammers (which is usually enough to filter out 90% or so) and to group sites together in informal co-operative spam-fighting networks like blogspam.net, so that once a spammer is spotted, they should get blocked on lots of sites (which blocks a bit over half of the remaining 10%).

Sadly, the rest gets shown to humans for decision. Real comments are so few and far between now that we really don’t want to risk turning real people away and killing discussions.

We used to go after spammers who got shown to humans, but there are now too many spammers and too many service providers who won’t kick spammers off their services: the spammers pay them and we don’t: all we could do was waste their money in support, so they stopped offering any support to non-customers. Is that a flaw in the co-operative nature of the Internet? Can we overcome it? Wish I knew…