Keith Moore wrote:
but in most cases this is inherent in the nature of the feature, and
has nothing whatsoever to do with the message format. if you could
add the feature at all, you could do so without changing the message
format and without the disruption that would cause.
This particular discussion is in relation to envelope and protocol
extensions, which has nothing to do with the *current* message format. The
point of this particular discussion is that a message format which
provided envelope data as a separate blob would allow the envelope
extensions to survive blackhole hops.
DSNs with NOTIFY=SUCCESS are an obvious example. The only thing I care
about is that delivery succeeds, meaning that this is a negotation between
the originator and the delivery server. The submission server and any
relays in between are completely irrelevant, yet due to the current
envelope-exchange design, they dictate the success of that feature.
OK, so let's do the case analysis for this one. Suppose this is done by a new
envelope extension defined after deployment of an infrastructure that doesn't
lose unrecognized extensions along the way. But wait a minute -- if this is a
new extension that's added after initial deployment an old delivery agent won't
recognize it either. So all of a sudden your notification of successful
delivery has become unreliable and dependent on an infrastructure upgrade of
delivery agents. This is worse than the present DSN scheme, where you always
get something back saying what happened.
The way you solve such problems is well known, of course: Criticality flags.
You add a flag to each extension to indicate whether or not recognition of it
is required for proper message handling. But unless you want this to revert to
being equivalent to the old DSN scheme you immediately need multiple
criticality flags -- in this case the extension must be recognized for delivery
and nowhere else. Pretty quickly you arrive at something where you'll have
submission, relay, and delivery criticality at a minimum. This gets to be
complex given that the boundaries between the operations tend to blur over
time.
Regardless, you've just reinvented an aspect of X.400, which had criticality
flags galore. And experience with X.400 taught us some interesting things about
criticality flags: Requirements that say criticality must be honored tend to
get ignored. Because of this extensions based on them being honored tend to
work quite poorly in practice.
Here's why. Suppose a get an extension I don't recognize that's marked as
critical for whatever operation I'm performing. I can either ignore it or honor
it. If I honor it I have to return an error to somebody somewhere, typically by
bouncing the message. This tends to get people pretty seriously upset. They
don't like it when mail bounces for reasons they don't understand and don't
care about and they don't like it when they get error messages they don't
understand.
The alternative of delivering the message, however, at least gets the mail
through, albeit while violating the standard. Now, it may be encrypted or
improperly formatted or otherwise damaged by the failure to honor the flag, but
this tends to get blamed on the message originator, not on the transport
process.
I'm a many year veteran of X.400 implementation and deployment, and I can tell
you this doesn't play out well at all in practice. As ad-hoc extensions get
developed and deployed (and they will no matter what the rules for such things
are) and cause disruption in message flow implementations will do what is
necessary to succeed. At most implementations will end up having a setting to
operate in standards compliant mode, and it won't be set very often.
And even if this plays out differently than it did in the X.400 space and the
bulk of implementations honor the flags, you still have a problem. A delivery
agent that doesn't recognize your NOTIFY=SUCCESS is still going to return an
error of some sort to someone. It isn't going to return something that's
recognizable as a delivery notification no matter what. So the result is that
the extension is inherently unreliable in a way that the current DSN extension
is not. And the way you fix that is by making the extension critical at every
step so you can detect where the failure is going to occur along the path and
return a sensible indication of what happened. In short, you end up reinventing
the current DSN extension even though your new infrastructure is more capable
in some ways than the present one.
You are correct in that this feature and others which need to survive
blackhole hops do not require a new message format by themselves. After
all, these are envelope extensions, which do not *currently* use the
message format directly.
Actually, an (admittedly complex) single SMTP extension to allow for envelope
elements that can pass through relays that don't understand them could be
defined now. You don't need a new message format for that. The reason this
hasn't been done is because when you actually start looking at the problem
space closely it hasn't been justifiable to do this.
However, another critical aspect of what I am talking about is that the
transer headers be separated from the message headers and moved into the
envelope blob. That *does* require a new message format. Not only is the
separation of transfer and message headers needed for basic features (such
as header encryption)
Header signatures and encryption can be accomplished by using nested messages.
Once again a new message format isn't required.
but the recombination of transfer headers with
envelope data is also necessary for other basic features (transfer header
survivability). So no, I don't think you can add new features without also
modifying the message format.
This is another place where lessons learned from X.400 tell us the space isn't
shaped the way you think it is. I really don't want to get the reasons why this
is so now, however. I've written too much as it is.
Ned