[Top] [All Lists]

Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting

2007-06-17 18:43:35

Hi all,

First, sorry for the (possibly - probably - undue) length of this note!

This thought has been bugging me for a while now, and I think it might be 
the time to let you ponder over it too, what with my needing sleep sooner 
or later and so on.  I especially want to know if the idea is a bad one 
simply because it might be a little bit dishonest; it relies on SMTP 
client behaviour being almost always predictably simple.  The extension 
needed to implement the idea will be a little different than most, even if 
most clients won't notice a thing while they are being treated to its 
compulsory effects.  So here goes:

Greylisting ( ) would be a neat trick but for 
one essential problem that is overlooked too often: it makes the 
assumption that all mail transactions, having been issued at least once, 
are always issued again and again until the delivery is successful from 
the same host that made the initial attempt.  That is, the IP address of 
the connecting client is a part of the identification of that transaction, 
and changing it is impossible without triggering a new greylisting.  
Without the IP address as part of the greylisting process, of course, you 
lose most of the defense offered by it.  There are even Greylisting 
implementations which rely exclusively on the IP address, with quite a 
good success rating.

However, in environments where clustering, proxying, load balancing or 
gatewaying are used to share the load of distributing mail from an 
identical source, greylisting will delay mail for longer than is necessary 
if multiple attempts happen to be made by different hosts in a cluster.   
If there are enough hosts sending mail, and depending on the greylisting 
timeouts and the queueing timeouts set at recipient and sender of the mail 
respectively, and supposing every host that tries is uniquely chosen, it 
might even be possible that mail will fail to be delivered in the time the 
sender decides mail can wait for delivery in a queue.   The mail will then 
be returned to the sender as undeliverable for a transient reason, which 
most greylisting implementations obfuscate as general system faults.  
(Although it seems that this problem is already known well enough, no-one 
seems to have noticed any such returned mail and plenty of people are 
using greylisting now.)  There is also the fact that the delay is applied 
to a *transaction*, not a *connection*; spammers are not punished for not 
trying again, they simply don't get their mail delivered in some cases 
when they fail to come back and the few times when the mail does get 
delivered they have simply been lucky enough to make the attempt with all 
the right circumstances at a later time when greylisting wasn't being 
imposed rather than because they were conscientiously trying to deliver 
again.  Ideally, there would be a way to make the spammer suffer until 
greylisting were no longer needed for a time.

My idea is to replace greylisting with a connection-delaying technique 
that will make the SMTP client wait until we're certain it is genuine.  We 
differ from Greylisting only in how we determine that an SMTP client is 
genuine; greylisting uses the fact that the client will come back, and 
we'll use the fact that the client knows what continuation lines are and 
that it must stay silent while we send periodic lines in the HELO/EHLO 
response for a given time.  This isn't unlike teergrubing, except that we 
don't need dedicated hosts for this technique and we slow the connection 
down as soon as it's possible rather than while mail is being delivered.  
We must not allow the client to pipeline any commands before EHLO or HELO, 
and we must not allow the client to initiate a MAIL transaction until 
issuing EHLO or HELO and completing the challenge either (unless, at the 
server's discretion, the client has already "Proven" itself).  A client 
that waits ("Proves itself") for a specified duration in minutes (five, 
say), during which it is receiving five-secondly (for instance) "Stay on 
the line" notices, shall eventually be allowed to see service extension 
lines and continue the mail transaction, while its IP address is stored in 
a whitelist.  The client must not say anything (or have pipelined anything 
in advance) beyond the HELO/EHLO line, for otherwise it shall be banished 
in an administratively-defined fashion (probably just a 421 response 
followed by a connection closure, but there's nothing to stop you 
blacklisting, firewalling, or what have you; people sensitive about read 
buffers can just empty the buffer with nonblocking reads, issue a final 
"250 XTARPIT Sorry, matey, you've blown it!" and then reject every 
following line except QUIT with 503).  Whitelisted entries run down to 
zero in seconds, starting with some administratively-defined time (a 
month, say).

How I came up with the idea: For a time, I used to rely on the fact that 
most genuine clients could wait up to the RFC 2821-defined five minute 
read timeout.  Unfortunately (of course, there has to be someone, doesn't 
there?) not all hosts would wait even a minute.  Sendmail's greet_pause 
rule gave me the idea of trying long timeouts before the 220 greeting in 
the hope that spammers would drop connections in disgust, not for just 
avoiding HTTP proxies identified with pre-greeting traffic.  They did.  I 
was on Windows at the time with Mercury/32, so I wrote a proxy (in Tcl, 
naturally) to do the delay work, drop connections with pre-greeting 
traffic and log local or remote closures and then experimented with 
various timeouts.  I soon noticed that spammers would be gradually 
(surprise, surprise) willing to wait longer and longer, until I had one 
waiting the full 150 seconds (2.5 minutes) without closing.  From the 
addresses I saw connect, I'd assume they were zombie boxes.  Soon 
thereafter, it was what I saw most of the time.  By this time, I was 
losing connections from such prominently uninteresting services as, and so gave up the hope.  With this scheme, I hope to take away 
any final excuse any self-respecting SMTP client might have for giving up 
too early, by essentially not letting them time out.  I think it can work, 
and that we can set our timeouts as high as we like and still manage to 
exhaust spammer patience without severely impacting our mail deliveries 
or, should we fail to impact the spammers when many people are 
implementing this and spammers are willing to wait, to at least increase 
the risk that their mass-mailing tools will break under them on however 
many machines they are deployed upon.  Each waiting session to a mailer is 
a thread or process wasted, but only the spammers lose by not being 
patient - the rest are tied up only for one occasion in a set period.  
Spammers are, as we all know, lazy, and look for excuses to move on.  The 
one thing they really do have, though, is potentially limitless botpower; 
we have to put ourselves in a position where we can really challenge it.

Now, to the extension.  This is where we feel a bit dirty.  I hope you can 
see so far that there are no opportunities for delaying the client before 
he greets us (no, we can't tamper with the TCP send window, nor can we add 
continuation lines to the 220 greeting line without definitely stepping on 
very shaky ground [RFCs 821 and 2821 don't clearly provision for a 
multiline response that isn't sent in response to a command]), and there's 
no point in delaying the client once he's greeted us since our PIPELINING 
extension is there for a reason and we want every good mailer to use it.  
(We might do this tarpit at the MAIL command if we didn't send PIPELINING 
to a new client and did to a known-good one, but then the first 
transaction of a new client would always have to be non-pipelined and if 
it happened to be, say, the output of a mailing list job for a given host, 
we'd have a lot of waste.  However, if the client is using HELO he can 
never use pipelining anyway, so we can delay it until MAIL if anyone here 
thinks it would help at all.)  The extension must be repeated multiple 
times in the EHLO response.  This is the important bit (and the point of 
this long message).  The below conditions are found to be true with 

This is all for nothing if SMTP clients will not ignore unknown extensions 
by simply forgetting about them and expect all of the lines sent to be in 
one write!

Personally, I'm pretty sure this is true for *most* truely robust clients, 
but it's also true that this extension makes novel use of the response by 
outputting lines on a timer.  OpenBSD's spamd stutters characters on a 
line; however, I am aware of at least one modern implementation which will 
actually choke if the entire line is not sent in one write with CRLF, 
never mind the group of lines.

It may be that the extension used can convey additional data to the 
client, such as how long it'll have to wait, but it could just as easily 
(and this is probably better for security reasons) convey a human-readable 
string (see below for an example).

There's no way for me to experiment with this idea easily without writing 
my own MTA (something I was thinking of doing anyway, also mostly in Tcl, 
as it happens), as Sendmail, my current choice, can't possibly be extended 
without pretty heavy patching.  I might try to whip up a proxy, but it'll 
be tricky if I still intend to keep all the other service extensions and 
if I can get other people's ideas first that would go a long way.

Using the time-honoured convention of "C: " marking lines sent by an SMTP 
client and "S: " marking those sent by the SMTP service, here's an example 
of the whole thing in action, as I imagine it.  The client is connecting 
from the same address on two occasions.  Not all the possibilities are 
shown here, but that can wait.  Remember that the HELO command works in 
the same way as EHLO except that the extensions aren't listed (or, as 
stated above, it could work in the usual way and the tarpit could be 
delayed until MAIL if it helps since there can be no pipelining anyway).  
Also, a server might (but it isn't shown here) be enterprising enough to 
note the hostname in EHLO/HELO in the whitelisting along with the IP 
address, so that it can react differently when the same host issues 
another name on a reconnection.  I don't know if this is necessarily a 
good idea, although it shouldn't affect most clients.  Here's the example:

S: [listening on SMTP port]
C: [Connects to server's listening port]
C: ehlo
S: Okay, I'll run off and hide.  You wait.
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [... and so on until five minutes have passed]
S: 250-XTARPITTING No, you're nice! (PURR)  Feed me!  I support:
S: 250-SIZE
S: 250-DSN
S: 250 HELP
C: [Goes on to complete any number of MAIL/RCPT/DATA|BDAT transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [After arbitrary amount of time prior to whitelist record expiry, 
connects again]
C: ehlo
S: You again? (PURR)  Feed me!  I support:
S: [ ... service extensions as before]
C: [As before, transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [closure of channel]

I think that's everything.  Please let me know what you think.  Is it 
feasible?  What would the implementers say?  Are the assumptions I've made 
bad or violating anything?  Even if it worked in practice (I'm quite sure 
it will), should it be discarded for being possibly a little too dependent 
on common implementation strategy (I.E. the robustness principle)?


Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: feedme(_at_)yamta(_dot_)org
Phone: +44 20 88008915
Mobile: +44 7986 053399