[Top] [All Lists]

Re: My greatest fear

1991-08-24 08:44:35
Mark (and to some degree, Ned),
  I share, obviously, the frustration, partially because I think we are 
very close.
  Let me take off from part of Ned's argument and suggest that we may be 
in much better shape than it might appear by doing a little failure 
  It seems to me that the concept that, if you can't present something 
(video on teletypes, etc), well, you can't present it, is key to lots of 
this and something that we have been periodically losing sight of.  It 
implies, for example, that, if I send a sound-and-light show off to a 
general purpose mailing list, lots of the messages are not going to get 
properly delivered and presented, and lots of the recipients are going 
to get irritated (with me, or with the list, or with someone else).  And
that is independent of either transport issues or message format--
nothing we can do short of buying *everyone* a multimedia terminal is 
going to solve it.
  Now, what happens when that message --again ignoring both format and 
transport issues for the moment-- arrives at the UA or whatever tries to 
manage presention to the user?  It cannot be delivered in the fashion we 
have normally thought of as "delivery".  Maybe we bounce it at the UA 
level.  Maybe we display all that nice transport-encoded gobblygook on 
the teletype and pretend that we have delivered it.  Maybe we encourage 
the user to write it to a file in the hope that, someday, the tooth 
fairy will donate a multimedia terminal and it will be possible to 
display the thing.  Those who believe in warning messages might send one 
out in these latter cases that would contain a translation of "delivered 
to user, but not presented; might never be".
  Maybe we do need some norms for what to do in these cases, although, 
traditionally, the Internet--designed for a kinder and gentler world in 
which email was used to carry only those things that we could presume 
everyone could read--has avoided discussing them.  Let's consider that 
part of a separate discussion, at least temporarily.
   For example, if I were writing specs for an RFC-XXXX-capable UA, I'd 
want to include a verb that would permit a *user* to force bouncing of a 
particular message, with text that said, e.g., "color video cannot be 
interpreted at this site".  And I'd like to see that go back with all of 
the "error message/bounce" apparatus, including null envelope return 
  [As an aside, for multipart messages, this implies a sort of line-item
veto, and one of the holes I see in RFC-XXXX is that the issues
associated with "I can/will accept part of that message, but not all of
it" have not been worked out at all.  The more we want MTAs to
"understand" RFC-XXXX and be able to fuss with it in transit, the more
important it is to start specifying the correct practices. ]
   The point is that there are some things are are not going to work 
(interoperate) and, again, no amount of diddling with protocols or RFCs
is going to change that. 

  OK, let's expand that reasoning a little bit.  Picking up from Mel's 
argument, if I try to open an 8bit connection to something that can't 
accept 8bit transport, it is going to reject the thing.  Period.  No way 
to change that as long as we insist on verb-negotiation (and the 
alternative is much worse).  Again, no amount of protocol-writing will 
fix that unless we can mandate an instant complete conversion, of which 
there is no chance.  As an extension of this, if we try to open an 8bit 
connection to an intermediate MTA such as a mail exchanger, no rule that
we can write and hope to "enforce" can prevent it from saying "I'm 
ultimately acting as the agent of the 7-bit-only mail server whose name 
is in the RCPT TO address; if you tried to open an 8bit connection to 
it, it would reject; so I am going to do so on its behalf".  We can try 
(possibly without success) to force that behavior on the intermediate, 
but trying to prohibit it is hopeless.
    Interestingly enough, in the direct virtual connection between 
originator and recipient MTA case, it doesn't make any difference 
whether the recipient machine understands RFC-XXXX (or any other 
extended message format): if it can't accept 8bit connections and hasn't
implemented the SMTP extensions, it is going to generate a fatal error
code in SMTP negotations and that is the end of the story.  Finis. 
Problem the originator--typically, in the real world, the originating
*user*, not some agent --is going to need to deal with. 
    That is all reality.  It is not a very nice reality, but it is, 
again, independent of any diddling we do with the protocols.  And I 
suggest that, if something cannot be delivered and presented, no matter 
what, it doesn't make a lot of difference where in the system it gets 
stopped.  I am not trying to build a case or model for system X knowing
the capabilities available to user A on system Y.  I am only suggesting 
that, if it does know, no amount of RFC-writing is going to prevent it 
from taking advantage of that information.  As someone (maybe, you, 
Mark) pointed out some days back, sending (or forwarding) mail that you 
know is going to be undeliverable is pointless, if not plain dumb.

   OK, so what is all the fuss about?  It is about one major case,
perhaps a second. 

Case 1: 8-bit, SMTP-extended host sending to 8-bit, SMTP-extended host 
over a multistep path that includes 7bit segments and where the 
intermediary at the first 8->7 boundary understands SMTP-8 and RFC-XXXX.
  (1a)  Despite that understanding, the intermediary refuses to get 
involved with this mess and rejects the transmission.  Case is just like 
those above, and vanishes.
  (1b) The intermediary at the other end of that 7bit path also 
understands SMTP-8 and RFC-XXXX and it and the first intermediary 
understand the relationships between themselves and with the destination 
host.  And they want to carry this.  This is a clean encapsulation case,
possibly the only one.  Almost regardless of what we tell them, they 
either use one of our protocols or invent a private one for carrying 
8bit data over the 7bit link and making 8bit data at the far end.  
Strictly an agreement between MTAs to use the link in a special way.  
Looks like end-to-end 8bit transmission to the outside world.  There is 
a special case of this in which the second intermediary turns out to be 
the delivery host.  As long as the two intermediaries keep their act 
together (which might be arbitrarily complex, but it is voluntary so who 
cares?), "we" still can't tell it from an end-to-end 8bit path. We don't
need to worry about this case, either. 
  (1c) The intermediary decides to accept the message, but to convert it 
into an network-acceptable 7bit form to pass it on.  The more complex we 
make the rules about the conversion it is expected to make, the more 
likely it is to just reject instead.  The more complex we make the rules 
about what the final delivery system (I make no MTA-UA distinction here) 
is expected to decode in this circumstance, the more likely *it* is to 
just reject instead.  This has the great potential for being, not a 
zero-sum game between MTA and UA responsibilities, but an "everyone 
loses and the message ends up back in the hands of the user who was 
naive enough to send it" story.  Probably there is a way out, but I 
haven't seen it yet.

  Conclusion about case 1: Subcases 1a and 1b are not IETF problems, and 
cannot be prevented.  Some things will bounce.  Some things will move 8 
bit mail transport transparently over 7 bit connections.  Subcase 1c is 
the source of a lot of our messes.  I think we can specify whatever we 
want without serious interoperability problems.  I think that, by 
inattention and better things to do with their time, a lot of 
implementations are going to turn it into case 1a.  And, the more 
complicated we "prove" that it has to be, the more will treat it that 
  [Aside 2: the only big risk in this reasoning is what gives Stef and 
me regular nightmares, which is that the folks who could get it right 
decide it isn't worth the effort, and the ones who don't understand, 
don't care, or think they know how to outsmart the rules implement and 
pollute everything they touch. ]
  [Aside 3: as Bob Smart has regularly pointed out, destination sites 
can mostly avoid case 1c from arising by an intelligent choice of MX 
hosts and preferences.  It might only arise if a user explicitly 
(source, percent, bang, or otherwise) routes something to a target host
via a pathological path or intermediary. Such users deserve whatever
they get. ] 

Case 2:  8-bit, SMTP-extended host sends a message, probably unwittingly, 
to a 7-bit-only host via an intermediary which can accept 8bit transport 
(but, of course, can't make a 7bit connection to the destination).
  (2a) Destination host does not understand RFC-XXXX and the encoding/ 
decoding rules either.  If the intermediary converts--and all of the 
"make it complicated enough and it will just bounce instead" comments in 
1c apply here--the end user is going to end up in the "garbage on
screen" case.  Some violence may be done to the principle that mail, 
once accepted, may not be not-delivered "for frivolous reasons". Some
users will be smart enough, and have ancillary tools enough, to decode. 
Most won't, and will wonder why "the system" is brain-damaged enough to
be delivering this garbage.  If one believes that RFC-XXXX will be
widely implemented, this may go into the "transitional situation.  too
bad. who cares?" category. 
  (2b) Destination host does understand RFC-XXXX and the encoding/
decoding rules, whatever they are.  The intermediary decides that the 
complexity of those rules, for it, are tolerable.  The destination host, 
upon receiving the message, decides that the complexity of the decoding 
is tolerable (if it doesn't, things bounce from there).

Since 1c can be eliminated by careful choices of MX hosts, I suggest
that 2b is the only case that is of real importance and that it takes a
large combination of things to even occur.  To review, they are: 
 (i) 7bit host destination host behind an 8bit intermediary.  If the 
intermediary were chosen to be 7bit, the message would be rejected at 
that point.
 (ii) intermediary willing to perform a conversion to whatever level of 
complexity we specify.
 (iii) destination host willing to deconvert whatever the intermediary 
sends it.

Now, if Mark, for example, decides to not implement nested decodings 
(we can't make him, and probably don't want to try) then this mail is 
going to bounce.  If the intermediary doesn't think the deep format 
understanding, rather than whole-message encapsulation, is worth the 
trouble, and we require the deep format understanding, then *it* may 
reject the message.

So, regardless of what we do, if there is going to be 8bit transport 
around, some messages are going to be rejected and/or bounce sometimes.  
And, because they could always find themselves directly connected to an 
"old" or "7bit only" host, every originating 8bit SMTP must be able to 
deal with rejections of 8bit transport.  I wonder how much energy cases 
2b and maybe 1c are worth?  Are people willing to trade them for this 
whole enterprise, if it comes to that?

Ned asks two questions (or I'm inferring two from his remarks) which 
I'd like to try to address, since I am feeling slightly less disgusted 
than Mark.

(1) Are all of the problems in the SMTP extensions? 
  Well, first of all, the problems are in the 8->7 conversion issues.  
There may be some additional problems in "binary" handling. They are not
in the 8bit transport of text, per se, at all. 
  Lots of the problems are in the conversion issues, but not all. 
RFC-XXXX raises, and has not yet addressed, issues all by itself having
to do with, e.g., un-presentability of messages and message parts and
what should be done about it.  If we take the attitude that network
responsibility ends end the bits arrive at the UA, then we immediately
escalate the need for delivery-to-presentation-agent acknowledgement,
since we can no longer make the claim that the network is sufficiently
reliable that silence (no rejection) is an adequate indication of
delivery to a place from which the user can read it. 
  It also contains forms and definitions that are just fine as a model 
for how to extend things beyond text, but are really very experimental.  
If I were the standardizing authority around here, I'd argue that 
RFC-XXXX has three parts:
  (i) A *model* for multipart, multimedia mail, including some header 
field names and abstract definitions.  This is standardizable to the 
extent that one can standardize models.
  (ii) A specific definition, header format, and protocol for text 
objects and for multipart objects consisting entirely of text objects.  
This is standardizable.
  (iii) One or more experimental protocols for "all that other stuff", 
to be treated as experimental until some solid experience down to the 
user level is accumulated.

Now, if we did things that way, we could design the UAs and the 8->7 
mechanisms with a clear idea of what we were handling as well as some 
insight into what might be coming.  The people who care most about 8bit 
transport and conversion could implement and experiment with converters
for the text and multipart-text cases.  By the time high-resolution 
moving pictures with sound emerged from "experimental", we might have
enough of an idea about how widely 8bit transport was going to be
available to write either "you must transport this over 8bit" or "you 
must transport this only over 7bit" rules for those forms, which would 
simplify things considerably.
   And I think that makes life a lot easier.

Are you seriously going to claim that the fact that you can specify extra
headers in messages (right now at the outermost level, later in inner levels)
is a "scoping gap you can drive a truck through"? Sure, you can specify
  I don't know what Mark was trying to claim, but I would suggest that 
the soft underbelly of RFC-XXXX is 822 itself, which is just too 
permissive about how extensions can go in.  You say that "Even X.400 
has a mechanism for specification of optional stuff...", which is 
exactly correct.  But, while X.400 "has a mechanism", 822 has proven, in 
practice, to have a declaration of "open season".
  You can't possibly know that a message format is RFC-XXXX unless you 
change the envelope and negotiate that.  Otherwise, it is just 
heuristics, statements about likelihood, and, ultimately, a leap of 
faith.  In retrospect, 822 should have required registration of every 
field name that didn't start with "X-".  It didn't, and we have no way 
to guess what is out there.

Without them you end up in the mess SMTP is facing now. Be
grateful for the things that are good about RFC822 here!
   Interesting observation.  I find 821, both in theory and in practice, 
much more tightly defined than 822.  When I go around the network 
bouncing EMAL verbs off servers they all reject it, neatly and 
consistently.  No exceptions so far.  With 822, we have fields floating 
around, none of them defined in standard or standards-track RFCs for 
such things as ICBM, Phone, Organization (is that a business, or the 
internal struture of the data?), Content-type, Errors-to, Warnings-to, 
Fax (the number of your machine, or the format it uses?), Character-set, 
Code,...  All valid extensions as 822 is written; a few of them threats 
to either RFC-XXXX or general email interoperability.
   The only "mess" SMTP is facing now is an attempt to make an 
extension/change and to make it both forward- (no problem) and backward- 
compatible (old system have to see what they expect, even if newer ones 
send out the new stuff).   We've got backward interoperability without 
problems: old clients can send to new servers, new clients can send to 
old servers, new-type messages are going to bounce neatly and without 
causing network or transport problems.  But backward compatibility, in 
which new clients can send new stuff to old servers and have the latter 
receive it as old stuff, is a *very* tough criterion, one which we 
rarely expect of anything.

There is a MAJOR difference between specification of some nonsensical header
that will be ignored and the introduction of headers that change the way
messages are broken down and parsed. RFC-XXXX is closed in regard to the
latter; it is open in regard to the former. The latter presents serious scoping
gaps; the former is a harmless detail.
   But 822 is not closed to any of this, and the very 822 properties 
that permit RFC-XXXX to come along and say "if you are going to read 
this particular message, then you need to understand these header 
fields in this way" permits another RFC, or a random implementor, to do 
the same.  Yes, this is mostly theoretical rather than real, or at least 
I hope it is, but I think it is quite dangerous to make robustness or 
"safe extensions" claims for RFC-XXXX without doing something about the 
822 problem/opportunity.


<Prev in Thread] Current Thread [Next in Thread>