[Top] [All Lists]

Re: Restrictions on the Content-Type "message"

1999-01-07 13:10:56
The way I understand it, application/news-transmission is a tunneling
protocol. As such, it's *requirements* are:

1. Upon unpacking from the tunnel, the articles in question must be bit-by-
bit *identical* to the articles that went in. That means redoing nested
encodings is Right Out[tm].

2. It being news, and news wanting no 7 bit restriction for very good
reasons (such as those being known to lead to flamewars all the time), the
mail restriction to 7 bit means that some sort of CTE is necessary here.

Now, the combination of 1 and 2 means there is no possible way to do this
without using nested encodings. OTOH, 1 also means that nested encodings
should not be a problem, since nobody is expected to *handle* nested
encodings - the mail and gateway parts only ever handle the outer
encodings, and other news software only ever sees the inner encodings.

There's nothing new here -- these are exactly the same arguments we heard 7
years ago when we first did MIME, and I understand them all too well. The point
you don't seem to be getting is that we considered all these choices way back
when MIME was designed, and we decided that we would not allow such
requirements to be met. Let me repeat that: We would not allow it. We
understood that situations would arise where such requirements would be given,
but we decided not to cater to them. So we wrote MIME with a no nested
encodings rule, and we didn't make an exemption for tunneling or anything else.

You can make up all the requirements you want. I can make up a requirement that
I want to be able to send packets will 50 billion bytes in them. But I cannot
do that with IPv4, and just because I've come up with a requirement to do it
doesn't mean that IPv4 has to change to be able to meet my needs. And it
doesn't mean I'll be able to get my protocol standardized when it requires such
a change in IPv4.

You're attempting to go back on a decision that was made some time ago, rightly
or wrongly, and which implementations have gone along with in good faith, and
which has resulted in a situation where if you're allowed to proceed along the
path you've chosen the result becomes untenable.

Now, the IETF has no formal rule that says one standard cannot unmake a rule
made in another standard. And there's no formal rule that says one standard
cannot change things that break old implementations. So in theory you can
register a new type that does these sorts of things to MIME.

But there is a formal rule that things have to interoperate. You cannot proceed
down the standards track in the presence of interoperability problems and you
cannot even get on the track if it can be shown that your proposal is
guaranteed to cause interoperability problems. Now, we all know that mail and
news are gatewayed all the time. Messages pass from news to mail and from mail
to news, sometimes multiple times.

So what happens when a application/news-message object passes from news to
mail, one which contains 8bit characters in the data, and the mail system
has to downgrade to 7bit? There are two choices:

(1) The mail system treats the object as opaque and slaps on a CTE. But then
    when someone tries to read this object using a mail reader, they see an
    opaque blob. The content, which likely contains perfectly reasonable MIME
    objects, is unreadable. The result is a failure to interoperate, one which
    would effectively prevent any sort of move to draft standard.
(2) The mail system treats the object as composite and encodes the inner parts.
    This puts the mail system in direct violation of your no-encoding rule.
    And a standard which has to be violated for things to work isn't acceptable.

Either way you have a violation of the intent, if not the letter, of the
standards process. And this is why I'm saying that the course you're on isn't

The long and short of it is that you need to give up on this idea of strict
tunneling. And once you do, a number of approaches open up that I think will
meet your real requirements without breaking things and which are viable as

The most obvious way to do this is to treat this as a gateway problem and
define the operation of mail <--> news gateways to get what you want. For
example, you can say that such gateways have to convert
application/news-message to message/rfc822; type=news and vice versa. Software
that adds and removes encodings is well understood, and should be no problem to
implement. It is especially easy if you deprecate use of quoted-printable and
base64 encoding of 8bit material in news (something you likely want to do
anyway). In such a case the gateway can simply remove quoted-printable and
base64 encoding of 8bit material willy-nilly; it need not consider what the
original encoding was. The result should be that headers are, modulo folding,
always returned to their original state and that bodies and their encodings are
preserved bit-for-bit. And you can further refine this with additional
restrictions, such as saying that boundary markers and such have to be
preserved, header lines cannot be moved (MIME actually says this already),
whatever.  You're never going to get to the point where exact bit-for-bit
compatibility is achieved, but there was never any chance of that happening
with mail transport of compositive MIME objects anyhow, no matter what the
standards say or how you label them, and this is something you're just going to
have to live with. (And if this causes problems for security services, well,
there are solutions to this that don't require bit-for-bit preservation of
content, which I'd be happy to elaborate on if need be.)

And once you do this, the rationale for having a new type vanishes. All you
will be doing is restricting what sorts of operations news systems can perform
on message/rfc822; type=news. But that is perfectly acceptable -- any standard
can at any time select a consistent subset of another standard.

Now, the case of application/news-message is not quite so simple. The way
I understand it, this one is actually meant (among others) for situations
where software would need to act on these nested encodings.

*However*, it still runs up against the problem that

a. News is meant to be 8 bit clean (not binary clean, though)

b. News is built around the assumption that, once a news article is
underway, most of the article headers and all of the body are safe from
change, in stark contrast to the mail case. (Typically, the exeptions are
the Path: header and the Xref: header.) There are applications around
*right* now that rely on this.

Obviously, a means that the headers of a news message, if sent by mail,
may still need to have a CTE applied. And b means that redoing CTEs is
still a problem.

So, it seems to me that we have here a case where the design goals of the
news communbity are actually incompatible with the design goals of the
MIME community, and there is no technical solution that will completely
satisfy both sides.

I disagree. I don't think you've explored all the options for moving forward. I
think that you've picked a way forward that solves one set of problems but not
a bunch of others (you've admitted your solution doesn't handle the outermost
header of a news message moving through mail), and moreover it causes lots of
new problems, many of them of a sort that are going to give you a great deal of
trouble in the standards process. But you've gotten hung up on how this one
solution works and made the conditions of it absolute. I think that with a bit
of compromise on your part we'll have a way to move forward that works and
which satisfies the requirements of both communities. It likely will dump
on gateways, but hey, those of us who write gateways are used to it.