[Top] [All Lists]

Re: Pipelining vs. multiple failure codes

2005-02-02 07:58:53

On Wed, Feb 02, 2005 at 01:44:09PM +0000, Richard Dawe wrote:

On Wed, 2005-02-02 at 11:13, Tony Finch wrote:
On Wed, 2 Feb 2005, Richard Dawe wrote:

The RFC on pipelining (RFC2920
<>) doesn't discuss what to
do on a temporary failure on, say, MAIL FROM.

The pipelining spec is hopelessly wooly.

Maybe it's worth doing an update to the spec, to make it clearer?

I found it very clear in overall, but at the same time
very challenging to implement efficiently and correctly
at the client side.

More so as I wanted to present some resemblance of context
in error reports, but not present _all_ of the context e.g.
50 RCPTs of which one is failing does not need to know about
49 successfull ones.

Server side is trivialish to implement even without buffering
optimizations ( = flush reply buffers to network only at sync-
points )  and fairly simple with them.  Flush replies always
when the input command stream is empty at the start of input
verb line collection.  Let them accumulate otherwise.  Plus
explicitely flush at PIPELINING-defined explicite sync points.
(just in case.)

The need for PIPELINING is marginal in all cases except when
you are sending large number of messages with large number of
recipients per message to a destination system which is far away,
and doing all that thru single connection.   At the time when
that happened, we already had fast enough international pipes,
but round-trip sync took some 0.5 seconds per recipient, and
RFC 821 half-duplex SMTP verb handshake waits took 99+% of time.
(Light speed is still finite, no matter what we try to do...)

Consider the following sequence:

  5xx (no MAIL FROM => RCPT TO not valid)
  5xx (no recipients)

What should the SMTP client conclude in this case? Should it bounce the
message or try again later?

It should try again later. The client should interpret the responses as if
it hadn't pipelined the commands. The first error is the important one.

Good, that's what I was hoping.

When my client is in pipelining mode, I collect all replies into
a client internal buffer space, and analyze them at next sync-point.
(I collect also all sent protocol lines there along with some meta-
data e.g. pointing to recipient data.)

At the sync-point I collect a bitmap of:


from received replies  (each reply is expected to be of some specific 
type, multiline processing including) and at the end of the sync-point
( = "DATA" reply collection ) an analysis is done to see the overall 
score:  MAIL_500 -> fail, MAIL_400 -> tempfail, (complex RCPT-fail things) 
DATA_OK -> ok ( + send the message ) ...
For each RCPT_500 an error report is generated (but only when MAIL_OK
was seen,)  for  each RCPT_400 a temp-fail report is generated (again
MAIL_OK exception.)   For RCPT_OK's a successfull RELAYING may get
reported after the message is sent.

Remote may report DATA_400 or DATA-500 even when all else has been OK.
Those do need their own processing as well.  (And "dots", too..)

Things do get even more contorted, when CHUNKING is added into the mix.
(Basically: your sync-point is at the end of the BDAT .. LAST chunk.
It also means, that BDAT command must _always_ glob up data amount
given in its parameter, and complain afterwards about petty things,
like lacking preceding MAIL+RCPTs..)

Ultimate optimization would be to do reply gathering and analysis
while transmit is under way so that if sender hears MAIL_400 or MAIL_500,
it can quit sending any more RCPTs or DATA/BDATs..  or if the sender
has  ran out of RCPTs without any being successfully accepted, or ...

(Outlook gets this grievously wrong and will uselessly report to users
"Valid RCPT command must precede DATA". Not advertising PIPELINING doesn't
help because it pipelines anyway.)

I wish all Outlooks would report even that much...

Lotus Domino also appears to break. I'm not sure if this is all versions
or just a particular version. I have no information about the version
number in the case I've seen.

Are there any other MTAs that have this problem?

Probably.  This PIPELINING is a very complex can of worms, where it
comes to SMTP client state-machine.

Thanks, regards,
Richard Dawe
Software Engineer - Mail Engine
MessageLabs Ltd., UK

/Matti Aarnio   <mea(_at_)nic(_dot_)funet(_dot_)fi>