Why you shouldn't rely on DKIM, SPF, and other whitelists
On the mail servers I manage we get thousands of spams every day. It's the same for every email server out there I guess, but there is something really different in France compared to other countries. In France the spam is actually legal and backed by the government. Yes a spam company can actually buy a database containing e-mails for non-profit organizations and companies from the INSEE public institute and then sell it again to its customers. The customers can then send spams to those email addresses, as long as they provide an opt-out unsubscribe link in the mail.
You can try and click on any number of opt-out links you wish, it doesn't change anything. At best you won't receive new spams from this specific announcer, but the spam provider often sells its database to hundreds of other announcers and will continue to relay spam through to you. When I actually tried to click as many opt-out links I could in a month, I actually got a huge increase of spam afterwards. It may seem incredible, but you can not trust those spam providers to actually stop sending spams. Weird. (Irony.)
There is a non-profit organization called "Signal Spam" which is supposed to fight spam, but a vast majority of its partners are actually spam providers like MailChimp or eCircle. Even the president of Signal Spam is the CEO of an « emailing » company called ExactTarget. I discuss the matter in details in this post about Signal Spam (in french). To this day Signal Spam doesn't seem committed at all to bring spam providers to justice or to lobby for a more protective law against spam.
About DKIM and SPF
A few years ago when I heard about SPF, and then about DKIM, I thought that would be a great idea to help fight spam. Now I can truly say that they are both very bad ideas. Even if they were actually working with real emails, which they don't, as they break most mailing lists and forwardings. But the worst is that out of the two thirds of emails my SMTP servers received who were using DKIM or SPF, more than half of those emails were actually spams.
Yes, spammers do use SPF and DKIM, and they do use them well. For this means that if an email does actually have a valid DKIM or passes the SPF test, there is more than a 50% chance that it is not a real email.
So when you are writing your spam filtering rules you should never trust that a message that passes the DKIM or SPF test is a legit email as it is often not the case.
The default rules for SpamAssassin for example are too not too optimistic if the mail passes the DKIM or SPF test, which is the right thing to do.
About whitelists
One thing that SpamAssassin didn't get right is that it trusts the whitelists. For example a valid DKIM domain which is in the SpamHaus Whitelist will get its spam score decreased by 3.5 points. That's a lot actually, as many spammers do manage to get listed in whitelists. One example is MailChimp, one of the largest spam provider out there, which is listed in ISIPP IADB.
As a result spams coming from MailChimp will get through your spam filter. What a shame.
I suggest to use those rules in your SpamAssassin local.cf file:
score DKIMDOMAIN_IN_DWL 0 score __RCVD_IN_DNSWL 0 score __RCVD_IN_IADB 0 score RCVD_IN_RP_CERTIFIED 0 score RCVD_IN_RP_SAFE 0
This will disable the whitelists completely, so that no request is made to those servers. It will save you some bandwidth and CPU, as well as rejecting more spams.
To help block more spams I will soon release a new DNS blacklist server which will help to reject emails from MailChimp, eCircle, MailJet and other "legal" spam providers.
Nicolas
The title of the article is quite misleading, IMHO it should be "Why you shouldn't rely on DKIM, SPF, and other whitelists to fight against commercial emails".
DKIM and SPF mainly protect from one thing: people who pretend to send emails from a domain where they are not actually entitled to do so.
It's obvious that announcers can have valid SPF and DKIM as they often send their mails from a domain of their own. SPF and DKIM were never meant to fight against that.
DKIM and SPF are actually quite good at fighting another kind of spam: phishing.
Nono
Totally agree with Nicolas.
It's a misunderstanding of what DKIM and SPF really are.
BohwaZ
Nope it is not. It's just that I didn't elaborate on the problems with DKIM or SPF about why they just don't work for the task they were meant to achieve. Yes they don't even solve the problem they were designed to solve (btw they are only marginally useful against phishing), and they create even more problems. One of them is the fact that it helps spammers deliver their messages, and that's the subject of this post.
Laurent
DKIM and SPF are not meant solely by itselves to aim for the whole spam issue. You seem like to assume both are the panacea/miracle againt spam. But they are not.
Both started with the purpose to move thing around the non-existent security of mail (lack of auth, lack of proof of integrity, clear-text no encryption, anyone can forge a mail)
They are not really meant to fight phishing. Not entirely. You can still phish with unofficial mail and still looks like too much real for the average net-user.
It prevents one third party from usurpating your domain -if SPF-aware and SPF_rule publish in DNS- and also prevents one third party to tamper the data (and thus prevents from inserting phishing links)
I agree with "they create even more problems", but only because SPF and DKIM mechanisms are complex to implement correctly, and let alone are not used widely.
The purpose of SPF is to authenticate the IP of the SMTP server who send you mail (spam or ham). It uses DNS mechanism (but DNS is not entirely secure-proof, this means, the security of SPF "equals" the security of DNS)
Since SMTP is TCP-based, one cannot usurpate easily an IP address (unless that person have a specific router) because of the three-way handshake.
If you are the owner of example.com, you can send mail from sender@example by SMTP relay. But so does everyone.
By using DNS text, you can say "Hey, my SMTP servers are A.B.C.D and W.X.Y.Z, beware of others IP who send you from @example.com"
If your SMTP server is SPF-aware, it will receive an incoming mail from an unknown IP source E.F.G.H and from sender@example.com and check for a reverse DNS lookup E.F.G.H and SPF rule @example.com. If it don't match, you can do whatever policy you want.
The purpose of DKIM is to make sure both data and mail headers are not tempered by anyone.
It also uses DNS mechanism and asymetric cryptography. It provides two hashed generated from the private keys. One for the data, One for some header fields + data.
A DKIM-aware server will check those hashes with the public key. If those don't match with the calculated ones, likewise SPF, you can apply whatever policy to those mails.
Both SPF and DKIM have their flaws (forwarding issue, so few people uses both SPF/DKIM rendering those checks to inexistant etc.)
In addition, DMARC tries to tie those two together and add some feedback between SMTP-domain and more DNS-published rules to follow or not and much more complexity and issues.
When I actually tried to click as many opt-out links I could in a month, I actually got a huge increase of spam afterwardsTo click in an unsollicited mail from a known or unknown companies = "Hey, it is my real address, please send me more spam"
Try to sue them? Can you bring a proof ?
If someone told you "SPF + DKIM + SpamAssassin = No more spam", you can sue that person. It it way much complicated than that, especially when you have strong constraint (like zero ham filtered)
To this day Signal Spam doesn't seem committed at all to bring spam providers to justice or to lobby for a more protective law against spam.To fight spam with justice, you must define spam precisely. You must have one legal definition first before sueing. But, what is a spam ? Can you define objectively?
So yes, you shouldn't rely on SPF by itself. It does not aim to address spam issue globally.
So yes, you shouldn't rely on DKIM by itself. Same remark
So yes, you shouldn't rely on SpamAssassin by itself. Not sufficient, beware of regexp (Scunthorpe problem)
So yes, you shouldn't rely on Bayesian anti-spam by itself. You should have ham and spam sources or let the program learn before - beware of multilanguage, images and so on...
So yes, you shouldn't rely on Whitelisting/External Blacklisting/Greylisting by itself. One friend can be tomorrow ennemy and also the reverse. Spammers can now wait 5min through the greylisting.
Good luck with spam though.