Hi all,
First, sorry for the (possibly - probably - undue) length of this note!
This thought has been bugging me for a while now, and I think it might be
the time to let you ponder over it too, what with my needing sleep sooner
or later and so on. I especially want to know if the idea is a bad one
simply because it might be a little bit dishonest; it relies on SMTP
client behaviour being almost always predictably simple. The extension
needed to implement the idea will be a little different than most, even if
most clients won't notice a thing while they are being treated to its
compulsory effects. So here goes:
Greylisting ( http://www.greylisting.org/ ) would be a neat trick but for
one essential problem that is overlooked too often: it makes the
assumption that all mail transactions, having been issued at least once,
are always issued again and again until the delivery is successful from
the same host that made the initial attempt. That is, the IP address of
the connecting client is a part of the identification of that transaction,
and changing it is impossible without triggering a new greylisting.
Without the IP address as part of the greylisting process, of course, you
lose most of the defense offered by it. There are even Greylisting
implementations which rely exclusively on the IP address, with quite a
good success rating.
However, in environments where clustering, proxying, load balancing or
gatewaying are used to share the load of distributing mail from an
identical source, greylisting will delay mail for longer than is necessary
if multiple attempts happen to be made by different hosts in a cluster.
If there are enough hosts sending mail, and depending on the greylisting
timeouts and the queueing timeouts set at recipient and sender of the mail
respectively, and supposing every host that tries is uniquely chosen, it
might even be possible that mail will fail to be delivered in the time the
sender decides mail can wait for delivery in a queue. The mail will then
be returned to the sender as undeliverable for a transient reason, which
most greylisting implementations obfuscate as general system faults.
(Although it seems that this problem is already known well enough, no-one
seems to have noticed any such returned mail and plenty of people are
using greylisting now.) There is also the fact that the delay is applied
to a *transaction*, not a *connection*; spammers are not punished for not
trying again, they simply don't get their mail delivered in some cases
when they fail to come back and the few times when the mail does get
delivered they have simply been lucky enough to make the attempt with all
the right circumstances at a later time when greylisting wasn't being
imposed rather than because they were conscientiously trying to deliver
again. Ideally, there would be a way to make the spammer suffer until
greylisting were no longer needed for a time.
My idea is to replace greylisting with a connection-delaying technique
that will make the SMTP client wait until we're certain it is genuine. We
differ from Greylisting only in how we determine that an SMTP client is
genuine; greylisting uses the fact that the client will come back, and
we'll use the fact that the client knows what continuation lines are and
that it must stay silent while we send periodic lines in the HELO/EHLO
response for a given time. This isn't unlike teergrubing, except that we
don't need dedicated hosts for this technique and we slow the connection
down as soon as it's possible rather than while mail is being delivered.
We must not allow the client to pipeline any commands before EHLO or HELO,
and we must not allow the client to initiate a MAIL transaction until
issuing EHLO or HELO and completing the challenge either (unless, at the
server's discretion, the client has already "Proven" itself). A client
that waits ("Proves itself") for a specified duration in minutes (five,
say), during which it is receiving five-secondly (for instance) "Stay on
the line" notices, shall eventually be allowed to see service extension
lines and continue the mail transaction, while its IP address is stored in
a whitelist. The client must not say anything (or have pipelined anything
in advance) beyond the HELO/EHLO line, for otherwise it shall be banished
in an administratively-defined fashion (probably just a 421 response
followed by a connection closure, but there's nothing to stop you
blacklisting, firewalling, or what have you; people sensitive about read
buffers can just empty the buffer with nonblocking reads, issue a final
"250 XTARPIT Sorry, matey, you've blown it!" and then reject every
following line except QUIT with 503). Whitelisted entries run down to
zero in seconds, starting with some administratively-defined time (a
month, say).
How I came up with the idea: For a time, I used to rely on the fact that
most genuine clients could wait up to the RFC 2821-defined five minute
read timeout. Unfortunately (of course, there has to be someone, doesn't
there?) not all hosts would wait even a minute. Sendmail's greet_pause
rule gave me the idea of trying long timeouts before the 220 greeting in
the hope that spammers would drop connections in disgust, not for just
avoiding HTTP proxies identified with pre-greeting traffic. They did. I
was on Windows at the time with Mercury/32, so I wrote a proxy (in Tcl,
naturally) to do the delay work, drop connections with pre-greeting
traffic and log local or remote closures and then experimented with
various timeouts. I soon noticed that spammers would be gradually
(surprise, surprise) willing to wait longer and longer, until I had one
waiting the full 150 seconds (2.5 minutes) without closing. From the
addresses I saw connect, I'd assume they were zombie boxes. Soon
thereafter, it was what I saw most of the time. By this time, I was
losing connections from such prominently uninteresting services as
Gmail.com, and so gave up the hope. With this scheme, I hope to take away
any final excuse any self-respecting SMTP client might have for giving up
too early, by essentially not letting them time out. I think it can work,
and that we can set our timeouts as high as we like and still manage to
exhaust spammer patience without severely impacting our mail deliveries
or, should we fail to impact the spammers when many people are
implementing this and spammers are willing to wait, to at least increase
the risk that their mass-mailing tools will break under them on however
many machines they are deployed upon. Each waiting session to a mailer is
a thread or process wasted, but only the spammers lose by not being
patient - the rest are tied up only for one occasion in a set period.
Spammers are, as we all know, lazy, and look for excuses to move on. The
one thing they really do have, though, is potentially limitless botpower;
we have to put ourselves in a position where we can really challenge it.
Now, to the extension. This is where we feel a bit dirty. I hope you can
see so far that there are no opportunities for delaying the client before
he greets us (no, we can't tamper with the TCP send window, nor can we add
continuation lines to the 220 greeting line without definitely stepping on
very shaky ground [RFCs 821 and 2821 don't clearly provision for a
multiline response that isn't sent in response to a command]), and there's
no point in delaying the client once he's greeted us since our PIPELINING
extension is there for a reason and we want every good mailer to use it.
(We might do this tarpit at the MAIL command if we didn't send PIPELINING
to a new client and did to a known-good one, but then the first
transaction of a new client would always have to be non-pipelined and if
it happened to be, say, the output of a mailing list job for a given host,
we'd have a lot of waste. However, if the client is using HELO he can
never use pipelining anyway, so we can delay it until MAIL if anyone here
thinks it would help at all.) The extension must be repeated multiple
times in the EHLO response. This is the important bit (and the point of
this long message). The below conditions are found to be true with
Sendmail:
This is all for nothing if SMTP clients will not ignore unknown extensions
by simply forgetting about them and expect all of the lines sent to be in
one write!
Personally, I'm pretty sure this is true for *most* truely robust clients,
but it's also true that this extension makes novel use of the response by
outputting lines on a timer. OpenBSD's spamd stutters characters on a
line; however, I am aware of at least one modern implementation which will
actually choke if the entire line is not sent in one write with CRLF,
never mind the group of lines.
It may be that the extension used can convey additional data to the
client, such as how long it'll have to wait, but it could just as easily
(and this is probably better for security reasons) convey a human-readable
string (see below for an example).
There's no way for me to experiment with this idea easily without writing
my own MTA (something I was thinking of doing anyway, also mostly in Tcl,
as it happens), as Sendmail, my current choice, can't possibly be extended
without pretty heavy patching. I might try to whip up a proxy, but it'll
be tricky if I still intend to keep all the other service extensions and
if I can get other people's ideas first that would go a long way.
Using the time-honoured convention of "C: " marking lines sent by an SMTP
client and "S: " marking those sent by the SMTP service, here's an example
of the whole thing in action, as I imagine it. The client is connecting
from the same address on two occasions. Not all the possibilities are
shown here, but that can wait. Remember that the HELO command works in
the same way as EHLO except that the extensions aren't listed (or, as
stated above, it could work in the usual way and the tarpit could be
delayed until MAIL if it helps since there can be no pipelining anyway).
Also, a server might (but it isn't shown here) be enterprising enough to
note the hostname in EHLO/HELO in the whitelisting along with the IP
address, so that it can react differently when the same host issues
another name on a reconnection. I don't know if this is necessarily a
good idea, although it shouldn't affect most clients. Here's the example:
S: [listening on SMTP port]
C: [Connects to server's listening port]
S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!)
C: ehlo goodmail.example.net
S: 250-bloodstone.yamta.org Okay, I'll run off and hide. You wait.
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [... and so on until five minutes have passed]
S: 250-XTARPITTING No, you're nice! (PURR) Feed me! I support:
S: 250-PIPELINING
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250-CHECKPOINT
S: 250-SIZE
S: 250-DSN
S: 250-AUTH CRAM-MD5 DIGEST-MD5 PLAIN
S: 250 HELP
C: [Goes on to complete any number of MAIL/RCPT/DATA|BDAT transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [After arbitrary amount of time prior to whitelist record expiry,
connects again]
S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!)
C: ehlo goodmail.example.net
S: 250-bloodstone.yamta.org You again? (PURR) Feed me! I support:
S: [ ... service extensions as before]
C: [As before, transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [closure of channel]
I think that's everything. Please let me know what you think. Is it
feasible? What would the implementers say? Are the assumptions I've made
bad or violating anything? Even if it worked in practice (I'm quite sure
it will), should it be discarded for being possibly a little too dependent
on common implementation strategy (I.E. the robustness principle)?
Cheers,
Sabahattin
--
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: feedme(_at_)yamta(_dot_)org
Phone: +44 20 88008915
Mobile: +44 7986 053399