[Top] [All Lists]

Re: Processing after the end of DATA

2010-08-09 22:39:36

John Levine wrote:

I believe this was a problem on CSNET when he wrote the RFC in 1988.
It is my impression that MTAs have gotten someone more robust in the
ensuing two decades.  Is it still a problem now?

From our experience and with customer support, generally only when scripts (and it can be more than one) all together take 3-5 minutes.

For a while I used a very aggressive greylister which soft failed
after the dot, kept a hash of the message, and wanted to see an
exact copy with the same hash to let the message through.  I stopped
using it, but not because of SMTP problems.  (A surprising number of
ESPs reconstruct the message rather than retransmitting it, so the
hash didn't match.)

Anyone got actual experience with delays at the dot?  It wouldn't be
hard for me to stick a module into my SMTP daemon that inserts extra
delays and see what breaks.

Since 2003, we had a flexible DATA scripting system designed specifically for operators or 3rd party developed local policy p-code scripts. They use an INI file to define the run order of the scripts. SMTP will pass every envelope data, connection data and the file name of the temporary spooled file received, and the script manager will run each script serially.

Each script can end with:

    PASS      - continue with next script
    FAIL      - stop scripts return 45x or 55x (DATA reject)
    FAIL_KEEP - same as FAIL, copy of file saved.

We provide two key stock scripts:

  SMTPFILTER-CHECKWORDS - list of Spam words to filter message.
  SMTPFILTER-WCGREY     - performs grey listing

Both are very popular and people share there list of words file. My personal spamword.txt has grow to at least 5000 words. It began as a very simple minded search word rejection script. You know grey listing.

What was learned and required code changes to address the issues.

The basic issue was the increase processing time when time delays as high as 3 or more minutes were possible, especially with the original simple search words program with no smarts.

So the first thing to speed the search word scripts and it was much supportive of MIME messages. What take 3-5 minutes was reduced to seconds.

But we also had to add socket disconnect detection to abort the scripts processing. We also added logic to check the read buffer for the QUIT command which is used to finally accept the message.

But aborting scripts also created the potential for a loop hole where a client will intentional abort a delayed DATA with the idea that its payload will be accepted.

So now the script system has EVENTS for disconnects and timeouts that will always process "required" scripts. So the operator has to design what order of scripts to run and which once should always regardless of drops and timeouts.

Finally, what I have seen is:

 - Client can drop, or

 - Clients issue QUIT, may wait and drop.

In both cases, I'm seen retransmissions but not always.

Overall, reducing the DATA processing tine is the key and keeping it shorter than 3 minutes (which is hardly seen now) is usually ok regarding client timeouts.

On a related note, in the past, we discussed KEEP-ALIVE concepts using continuation reply line every X time to keep the client alive. This was explored and proven to work with many modern clients. However, there are legacy clients that do not use the last "XXXspace" line, they use the first line only and the RFC 5321 specs was codified to suggest all continuation lines use the same reply code.


Hector Santos