On Wed, Mar 24, 2004 at 07:48:32PM +0000, Tim Meadowcroft wrote:
Unfortunately my current hosting company (Interland) won't let me add SPF
records while they control my DNS, so I'm still getting all the bounces, all
the more reason why I need to change to a hosting company with a smarter
attitude...
I'm hoping that as soon as I add SPF to my domain it'll cut all these bounces
and in turn eventually stop the joe-jobbing
It's very unlikely that you'll see a significant decrease in joe-jobs in the
short to medium term, since the vast majority of MTAs will ignore your SPF
records.
(last year I wrote a script that
traced the source IP of the original email and complained to the abuse
department of the appropriate ISP, but after 25,000 complaints led to only
about 10 replies, and the spammer started forging headers, I gave up on it).
And SPF won't help with forged headers one jot.
Along this line, I'm using qpsmtp with qmail for receiving my mail, and it
has
a module to support SPF. I'm going to suggest a patch to this so that if the
SPF check fails it shoudn't immediately refuse, but should instead sleep for
5-10 seconds before refusing - if we all did this it would have the effect of
tar-pitting the spammers so making non-deliveries even more expensive - this
is the equivalent of wasting the time of telemarketers rather than just
hanging up on them (which leaves them free to make another call immediately).
Unlike a telephone call, a TCP connection is free, apart from a few bytes of
memory used for a socket data structure. With 512MB DIMMs at about £50 in
the UK at the moment, it's not an issue. If they are connecting to different
destinations, then the 16-bit limit on TCP source port numbers is not an
issue either.
I know AOL has a whitelist that you can ask to be put on if you genuinely
send
opt-in large mailings, I just wish they had a "please blacklist me" too -
like a custom SPF list that you could notify them of...
My suggestion would be to choose a solution which fits the problem. If your
problem is that you receive lots of joe-job bounces, the solution is to use
SRS-style sender cookies in the envelope of your outgoing mail, as
documented on this list. It's an instant solution which doesn't require
cooperation of anyone else on the Internet.
I am only just catching up on SRS/SPF mail after being away for several
weeks, but I think I now know enough to know that it's not worth pursuing.
I am all in favour of modifying E-mail in such a way as to make spammers'
lives difficult, but SPF is not the way to go. In essence my conclusions
are:
(1) The cure is worse than the disease
Implementing SPF breaks perfectly valid and established practices, such as
mail forwarding, and risks losing legitimate bounces.
For me, the real reason that spam is such an annoyance is that it risks
losing legitimate mail in a sea of illegitimate mail; you end up having to
filter your inbox, whether manually or algorithmically, and this filtering
ends up accidentally deleting valid messages.
If you change E-mail in such a way that legitimate messages are lost, then
you are introducing another problem of the same order of magnitude as spam.
This is only going to be worth doing if it is a *strong* solution to the
problem, i.e. the final benefit will outweigh the cost, and (IMO) SPF is not
a strong solution.
(2) The disease will mutate
It's so trivial for spammers and virus writers to bypass SPF, that it would
happen almost overnight. When you send mail from exampleisp.com's IP address
space, send MAIL FROM:<anyrandomuser(_at_)exampleisp(_dot_)com>. Cleverer spam
and
virus writers will also use other domains which SPF permits from that IP
address (or which do not declare SPF information); such information is
easily located in the DNS.
This of course does not make spam any easier to trace. Anyone with half a
brain cell would be able to tell that the mail came from exampleisp.com by
looking at the Received: headers. The Return-Path: header is a red herring.
The From: header, which the vast majority of users see, is unaffected.
And it doesn't fix joe jobs - anyrandomuser(_at_)exampleisp(_dot_)com will
receive the
bounce.
(3) The cure is for the wrong disease
E-mail has arguably three huge diseases at the moment:
- spam
- viruses
- untrustworthiness / forgeability
The disease which SPF attempts to fix is "untrustworthiness of
envelope-sender information", which means the Return-Path: header on
delivered mail. In some instances it may check the EHLO name which appears
in a Received: header. Both are invisible to normal users. It does not add
any trust to the visible mail headers or body at all. It does not address
the issue of mail sent with a null envelope sender, apart from an
ill-advised check on the EHLO name.
So there's only one thing which it *might* fix, which is the receipt of
joe-job bounces. But it won't work until the whole world honours SPF records
*and* has modified their existing records to remove the '?all' from the end.
I doubt this will happen, and there are much quicker and simpler solutions
to joe-jobs anyway.
Even the trustworthiness of the envelope sender is very weak in SPF. SPF can
assert:
- this mail definitely could not be from bob(_at_)example(_dot_)com
(because this IP address is not entitled to use example.com)
- this mail *might* be from bob(_at_)example(_dot_)com
(because this IP address is entitled to use the domain; but maybe it's
joe(_at_)example(_dot_)com forging a mail)
But it can't assert that the mail *is* from bob(_at_)example(_dot_)com(_dot_)
(4) To have any effect it requires 100% deployment across the Internet
Which simply won't happen, because (a) there are enough smart people out
there who realise it's better not to do it, and (b) they are plenty of lazy
people who don't care.
(5) It gives no additional information to identify spammers
An open relay has an IP address. A spammer's netblock has IP addresses. What
else do you need?
Domains give you nothing else, and it's far easier to get fresh domains than
fresh IP addresses. A web-of-trust for domains simply won't work; if
spammers don't buy thousands of disposable domains, they will simply use
their ISP's domain [the spammer has no connection with that domain in the
registry, of course]. You then either:
- accept all mail from that ISP's domain (and accept the spam), or
- block all mail from that ISP's domain (in which case you might as well
just blacklist the whole ISP and be done with it)
And of course, spam sent with a null envelope sender isn't going to have
*any* information for SPF to validate, apart from the weak EHLO check, which
is another cure-worse-than-disease (as it has an extremely high risk of
rejecting valid bounces)
If SPF *did* encourage spammers to send more spam with null envelope
senders, then it would just encourage more ISPs to take the dreadful
decision to discard all incoming bounces. That would just make E-mail far
less trustworthy than even it is now. The cleverer ISPs will implement
sender cookies rather than just dropping bounces, but they can do that now
anyway.
(6) There is no upgrade path to a final solution
Suppose SPF is implemented everywhere, and the spammers have modified their
behaviour so that SPF doesn't inhibit them. SPF doesn't offer any further
path to a "final solution"; it's a dead end.
What would this final solution look like? I think:
- it would have to give immediate and increasing benefit to anyone who
implements it, without having to wait for 100% deployment across the
Internet.
- it would have to increase the trustworthiness of E-mail itself, in
particular that the From: header can be validated. Validating hidden headers
is of little benefit to anyone.
- it would give an upgrade path towards *complete* validation of the From:
header, that is, not only that the domain part is presumed valid but that
the full E-mail address is known to be valid.
- it would give sufficient additional benefits to be worth the cost of
implementation and any transitional pain. Such side benefits might include
strong cryptographic assurance that the *body* of the E-mail has not been
tampered with, and was written by the asserted sender of the mail.
- it could be implemented incrementally at the ISP MTA, without requiring
users to replace their MUAs.
As one option, such a solution might look something like a cryptographic
signature [*], signed by the ISP's smarthost, with their public key in the
DNS for anyone to check. The signature could assert that the user had
identified themselves with SMTP AUTH and include the user ID; or it could
assert that the domain belongs to this ISP; or it could assert only that the
message had been through this ISP and was received from a particular IP
address. External information in the DNS could assert whether messages
"From: bob(_at_)example(_dot_)com" are associated with a particular ISP, a
particular
IP address, or a particular SMTP AUTH user identity at a particular ISP.
Such a solution would be upgradable so that individual end users could sign
messages themselves, and put their own public key in the DNS (e.g. messages
From: bob(_at_)example(_dot_)com would be signed by the key found under
bob._at.example.com). This would fully free roaming users without having to
SMTP AUTH via their home mail server, and increase the level of trust that
the message had not been tampered with before leaving the client's machine.
Such a solution might be upgradeable to include encryption of messages, so
that it becomes a full public key infrastructure (e.g. if I find a public
key for bob._at.example.com then I can encrypt messages to him using that
key; if I find one for the ISP example.com then I can encrypt using that,
and the ISP will decrypt it before putting it in Bob's mailbox). Such
capabilities would of course be flagged along with the keys themselves.
Anyway, all this could transform the reliability and trustworthiness of
E-mail. Would it help control spam though? Well, I guess:
- a strong cryptographic assertion that a message came from ISP X would be
good evidence to report or blacklist ISP X; this would be easier and less
error-prone than parsing Received: headers
- a strong cryptographic assertion that a message originated from a known
trusted user or domain could be used as a whilelist key (i.e. it prevents
spammers guessing the names of our friends and colleagues and sending spam
which appears to be from one of them)
But otherwise, a spammer is a spammer. A spammer can generate many online
identities, so if we are going to accept E-mail from people we don't know,
then we are likely to accept spam. The only identity they can't easily
modify is their IP address, or the IP address of the proxies they relay
through.
We are still down to chasing spammers when we find them, which means
ultimately going through the courts. Received: headers give information
which can locate them (unfortunately, with dynamic IP and the absence of
SMTP AUTH it requires the cooperation of the spammer's ISP); that's exactly
the situation as it is now. Maybe a crypto solution which is stronger than
Received: can make this process more efficient and reliable.
And viruses? Well, they're basically automated spam. If we move to a world
where messages are signed and people are using SMTP AUTH, then we may be
able to tell exactly *who* sent us a virus, which may mean it can get
cleaned up more quickly. It's certainly not a cure though.
Incidentally, I would *not* be swayed too much by trivial concerns like "we
want to reject the mail before the DATA stage to save bandwidth"; any burned
bandwidth is costing spammers equally to us, and any *strong* solution is
going to generate benefits over the longer term. Sure, define ESMTP
extensions or even a whole new replacement protocol to fix these problems
properly, once and for all. If we can see that it's really a solution, and
there is a workable transition plan, then we'll move ahead with it. But if
it's just tinkering at the edges, then it won't be done.
If you've read this far - thanks, and thanks for providing an interesting
discussion for me to read. Time for me to sign off now though; the volume of
mail on this list is too much for me to deal with in addition to my spam and
viruses :-)
Regards,
Brian Candler.
[*] Alternatively: the final solution could be a rethink of how SMTP works.
One option might be to make recipients of mail connect to the sender to
*pull* the message. In other words, if bob(_at_)example(_dot_)com sends me a
mail, he
just sends me a message-ID cookie, and I follow the DNS information for
example.com to connect to his mailserver and pull the requested mail by
message-ID (like NNTP...). Effectively it's validating sender by callback.
I'm sure there are other radical solutions.