Mark (and to some degree, Ned),
I share, obviously, the frustration, partially because I think we are
Let me take off from part of Ned's argument and suggest that we may be
in much better shape than it might appear by doing a little failure
It seems to me that the concept that, if you can't present something
(video on teletypes, etc), well, you can't present it, is key to lots of
this and something that we have been periodically losing sight of. It
implies, for example, that, if I send a sound-and-light show off to a
general purpose mailing list, lots of the messages are not going to get
properly delivered and presented, and lots of the recipients are going
to get irritated (with me, or with the list, or with someone else). And
that is independent of either transport issues or message format--
nothing we can do short of buying *everyone* a multimedia terminal is
going to solve it.
Now, what happens when that message --again ignoring both format and
transport issues for the moment-- arrives at the UA or whatever tries to
manage presention to the user? It cannot be delivered in the fashion we
have normally thought of as "delivery". Maybe we bounce it at the UA
level. Maybe we display all that nice transport-encoded gobblygook on
the teletype and pretend that we have delivered it. Maybe we encourage
the user to write it to a file in the hope that, someday, the tooth
fairy will donate a multimedia terminal and it will be possible to
display the thing. Those who believe in warning messages might send one
out in these latter cases that would contain a translation of "delivered
to user, but not presented; might never be".
Maybe we do need some norms for what to do in these cases, although,
traditionally, the Internet--designed for a kinder and gentler world in
which email was used to carry only those things that we could presume
everyone could read--has avoided discussing them. Let's consider that
part of a separate discussion, at least temporarily.
For example, if I were writing specs for an RFC-XXXX-capable UA, I'd
want to include a verb that would permit a *user* to force bouncing of a
particular message, with text that said, e.g., "color video cannot be
interpreted at this site". And I'd like to see that go back with all of
the "error message/bounce" apparatus, including null envelope return
[As an aside, for multipart messages, this implies a sort of line-item
veto, and one of the holes I see in RFC-XXXX is that the issues
associated with "I can/will accept part of that message, but not all of
it" have not been worked out at all. The more we want MTAs to
"understand" RFC-XXXX and be able to fuss with it in transit, the more
important it is to start specifying the correct practices. ]
The point is that there are some things are are not going to work
(interoperate) and, again, no amount of diddling with protocols or RFCs
is going to change that.
OK, let's expand that reasoning a little bit. Picking up from Mel's
argument, if I try to open an 8bit connection to something that can't
accept 8bit transport, it is going to reject the thing. Period. No way
to change that as long as we insist on verb-negotiation (and the
alternative is much worse). Again, no amount of protocol-writing will
fix that unless we can mandate an instant complete conversion, of which
there is no chance. As an extension of this, if we try to open an 8bit
connection to an intermediate MTA such as a mail exchanger, no rule that
we can write and hope to "enforce" can prevent it from saying "I'm
ultimately acting as the agent of the 7-bit-only mail server whose name
is in the RCPT TO address; if you tried to open an 8bit connection to
it, it would reject; so I am going to do so on its behalf". We can try
(possibly without success) to force that behavior on the intermediate,
but trying to prohibit it is hopeless.
Interestingly enough, in the direct virtual connection between
originator and recipient MTA case, it doesn't make any difference
whether the recipient machine understands RFC-XXXX (or any other
extended message format): if it can't accept 8bit connections and hasn't
implemented the SMTP extensions, it is going to generate a fatal error
code in SMTP negotations and that is the end of the story. Finis.
Problem the originator--typically, in the real world, the originating
*user*, not some agent --is going to need to deal with.
That is all reality. It is not a very nice reality, but it is,
again, independent of any diddling we do with the protocols. And I
suggest that, if something cannot be delivered and presented, no matter
what, it doesn't make a lot of difference where in the system it gets
stopped. I am not trying to build a case or model for system X knowing
the capabilities available to user A on system Y. I am only suggesting
that, if it does know, no amount of RFC-writing is going to prevent it
from taking advantage of that information. As someone (maybe, you,
Mark) pointed out some days back, sending (or forwarding) mail that you
know is going to be undeliverable is pointless, if not plain dumb.
OK, so what is all the fuss about? It is about one major case,
perhaps a second.
Case 1: 8-bit, SMTP-extended host sending to 8-bit, SMTP-extended host
over a multistep path that includes 7bit segments and where the
intermediary at the first 8->7 boundary understands SMTP-8 and RFC-XXXX.
(1a) Despite that understanding, the intermediary refuses to get
involved with this mess and rejects the transmission. Case is just like
those above, and vanishes.
(1b) The intermediary at the other end of that 7bit path also
understands SMTP-8 and RFC-XXXX and it and the first intermediary
understand the relationships between themselves and with the destination
host. And they want to carry this. This is a clean encapsulation case,
possibly the only one. Almost regardless of what we tell them, they
either use one of our protocols or invent a private one for carrying
8bit data over the 7bit link and making 8bit data at the far end.
Strictly an agreement between MTAs to use the link in a special way.
Looks like end-to-end 8bit transmission to the outside world. There is
a special case of this in which the second intermediary turns out to be
the delivery host. As long as the two intermediaries keep their act
together (which might be arbitrarily complex, but it is voluntary so who
cares?), "we" still can't tell it from an end-to-end 8bit path. We don't
need to worry about this case, either.
(1c) The intermediary decides to accept the message, but to convert it
into an network-acceptable 7bit form to pass it on. The more complex we
make the rules about the conversion it is expected to make, the more
likely it is to just reject instead. The more complex we make the rules
about what the final delivery system (I make no MTA-UA distinction here)
is expected to decode in this circumstance, the more likely *it* is to
just reject instead. This has the great potential for being, not a
zero-sum game between MTA and UA responsibilities, but an "everyone
loses and the message ends up back in the hands of the user who was
naive enough to send it" story. Probably there is a way out, but I
haven't seen it yet.
Conclusion about case 1: Subcases 1a and 1b are not IETF problems, and
cannot be prevented. Some things will bounce. Some things will move 8
bit mail transport transparently over 7 bit connections. Subcase 1c is
the source of a lot of our messes. I think we can specify whatever we
want without serious interoperability problems. I think that, by
inattention and better things to do with their time, a lot of
implementations are going to turn it into case 1a. And, the more
complicated we "prove" that it has to be, the more will treat it that
[Aside 2: the only big risk in this reasoning is what gives Stef and
me regular nightmares, which is that the folks who could get it right
decide it isn't worth the effort, and the ones who don't understand,
don't care, or think they know how to outsmart the rules implement and
pollute everything they touch. ]
[Aside 3: as Bob Smart has regularly pointed out, destination sites
can mostly avoid case 1c from arising by an intelligent choice of MX
hosts and preferences. It might only arise if a user explicitly
(source, percent, bang, or otherwise) routes something to a target host
via a pathological path or intermediary. Such users deserve whatever
they get. ]
Case 2: 8-bit, SMTP-extended host sends a message, probably unwittingly,
to a 7-bit-only host via an intermediary which can accept 8bit transport
(but, of course, can't make a 7bit connection to the destination).
(2a) Destination host does not understand RFC-XXXX and the encoding/
decoding rules either. If the intermediary converts--and all of the
"make it complicated enough and it will just bounce instead" comments in
1c apply here--the end user is going to end up in the "garbage on
screen" case. Some violence may be done to the principle that mail,
once accepted, may not be not-delivered "for frivolous reasons". Some
users will be smart enough, and have ancillary tools enough, to decode.
Most won't, and will wonder why "the system" is brain-damaged enough to
be delivering this garbage. If one believes that RFC-XXXX will be
widely implemented, this may go into the "transitional situation. too
bad. who cares?" category.
(2b) Destination host does understand RFC-XXXX and the encoding/
decoding rules, whatever they are. The intermediary decides that the
complexity of those rules, for it, are tolerable. The destination host,
upon receiving the message, decides that the complexity of the decoding
is tolerable (if it doesn't, things bounce from there).
Since 1c can be eliminated by careful choices of MX hosts, I suggest
that 2b is the only case that is of real importance and that it takes a
large combination of things to even occur. To review, they are:
(i) 7bit host destination host behind an 8bit intermediary. If the
intermediary were chosen to be 7bit, the message would be rejected at
(ii) intermediary willing to perform a conversion to whatever level of
complexity we specify.
(iii) destination host willing to deconvert whatever the intermediary
Now, if Mark, for example, decides to not implement nested decodings
(we can't make him, and probably don't want to try) then this mail is
going to bounce. If the intermediary doesn't think the deep format
understanding, rather than whole-message encapsulation, is worth the
trouble, and we require the deep format understanding, then *it* may
reject the message.
So, regardless of what we do, if there is going to be 8bit transport
around, some messages are going to be rejected and/or bounce sometimes.
And, because they could always find themselves directly connected to an
"old" or "7bit only" host, every originating 8bit SMTP must be able to
deal with rejections of 8bit transport. I wonder how much energy cases
2b and maybe 1c are worth? Are people willing to trade them for this
whole enterprise, if it comes to that?
Ned asks two questions (or I'm inferring two from his remarks) which
I'd like to try to address, since I am feeling slightly less disgusted
(1) Are all of the problems in the SMTP extensions?
Well, first of all, the problems are in the 8->7 conversion issues.
There may be some additional problems in "binary" handling. They are not
in the 8bit transport of text, per se, at all.
Lots of the problems are in the conversion issues, but not all.
RFC-XXXX raises, and has not yet addressed, issues all by itself having
to do with, e.g., un-presentability of messages and message parts and
what should be done about it. If we take the attitude that network
responsibility ends end the bits arrive at the UA, then we immediately
escalate the need for delivery-to-presentation-agent acknowledgement,
since we can no longer make the claim that the network is sufficiently
reliable that silence (no rejection) is an adequate indication of
delivery to a place from which the user can read it.
It also contains forms and definitions that are just fine as a model
for how to extend things beyond text, but are really very experimental.
If I were the standardizing authority around here, I'd argue that
RFC-XXXX has three parts:
(i) A *model* for multipart, multimedia mail, including some header
field names and abstract definitions. This is standardizable to the
extent that one can standardize models.
(ii) A specific definition, header format, and protocol for text
objects and for multipart objects consisting entirely of text objects.
This is standardizable.
(iii) One or more experimental protocols for "all that other stuff",
to be treated as experimental until some solid experience down to the
user level is accumulated.
Now, if we did things that way, we could design the UAs and the 8->7
mechanisms with a clear idea of what we were handling as well as some
insight into what might be coming. The people who care most about 8bit
transport and conversion could implement and experiment with converters
for the text and multipart-text cases. By the time high-resolution
moving pictures with sound emerged from "experimental", we might have
enough of an idea about how widely 8bit transport was going to be
available to write either "you must transport this over 8bit" or "you
must transport this only over 7bit" rules for those forms, which would
simplify things considerably.
And I think that makes life a lot easier.
Are you seriously going to claim that the fact that you can specify extra
headers in messages (right now at the outermost level, later in inner levels)
is a "scoping gap you can drive a truck through"? Sure, you can specify
I don't know what Mark was trying to claim, but I would suggest that
the soft underbelly of RFC-XXXX is 822 itself, which is just too
permissive about how extensions can go in. You say that "Even X.400
has a mechanism for specification of optional stuff...", which is
exactly correct. But, while X.400 "has a mechanism", 822 has proven, in
practice, to have a declaration of "open season".
You can't possibly know that a message format is RFC-XXXX unless you
change the envelope and negotiate that. Otherwise, it is just
heuristics, statements about likelihood, and, ultimately, a leap of
faith. In retrospect, 822 should have required registration of every
field name that didn't start with "X-". It didn't, and we have no way
to guess what is out there.
Without them you end up in the mess SMTP is facing now. Be
grateful for the things that are good about RFC822 here!
Interesting observation. I find 821, both in theory and in practice,
much more tightly defined than 822. When I go around the network
bouncing EMAL verbs off servers they all reject it, neatly and
consistently. No exceptions so far. With 822, we have fields floating
around, none of them defined in standard or standards-track RFCs for
such things as ICBM, Phone, Organization (is that a business, or the
internal struture of the data?), Content-type, Errors-to, Warnings-to,
Fax (the number of your machine, or the format it uses?), Character-set,
Code,... All valid extensions as 822 is written; a few of them threats
to either RFC-XXXX or general email interoperability.
The only "mess" SMTP is facing now is an attempt to make an
extension/change and to make it both forward- (no problem) and backward-
compatible (old system have to see what they expect, even if newer ones
send out the new stuff). We've got backward interoperability without
problems: old clients can send to new servers, new clients can send to
old servers, new-type messages are going to bounce neatly and without
causing network or transport problems. But backward compatibility, in
which new clients can send new stuff to old servers and have the latter
receive it as old stuff, is a *very* tough criterion, one which we
rarely expect of anything.
There is a MAJOR difference between specification of some nonsensical header
that will be ignored and the introduction of headers that change the way
messages are broken down and parsed. RFC-XXXX is closed in regard to the
latter; it is open in regard to the former. The latter presents serious scoping
gaps; the former is a harmless detail.
But 822 is not closed to any of this, and the very 822 properties
that permit RFC-XXXX to come along and say "if you are going to read
this particular message, then you need to understand these header
fields in this way" permits another RFC, or a random implementor, to do
the same. Yes, this is mostly theoretical rather than real, or at least
I hope it is, but I think it is quite dangerous to make robustness or
"safe extensions" claims for RFC-XXXX without doing something about the