Re: SMTP compressed protocol...

--On Friday, 05 December, 2003 15:29 +1200 Franck Martin 
<franck(_at_)sopac(_dot_)org> wrote:
> While talking about HTML in e-mail messages that consume a lot
> of bandwidth...
>
> Why SMTP servers do not negotiate to send an 8bit compressed
> stream between themselves. The same way HTTP negotiate a
> compressed stream between client and server if the client has
> the capability...
When the relevant WG looked briefly at the question a _long_ 
time ago (in Internet years), the conclusion was that it wasn't, 
in general, worth the trouble.  Or, if you prefer, worth the 
trouble often enough to justify the effort.   That conclusion 
was conditioned, if I recall, by the combination of several 
things:
(1) At the time, there was no obvious compression algorithm 
(given IPR encumbrances, etc.) and standards-conforming 8bit 
transport, having just been defined, was obviously not widely 
deployed.   That second condition has obviously changed.
(2) Relaying complicates everything with SMTP, since one could 
not guarantee that a negotiation was going on between sending 
and recipient machines.  If one had to compress between sender 
and initial relay, then have the relay decompress (and maybe 
recompress using a different algorithm) in order to pass the 
message along, it might cancel out any possible efficiencies.
(3) Some message body parts (usually "attachments"), especially 
large ones, are already compressed (e.g., zip files or 
equivalent) or not effectively compressible (e.g., almost 
anything encrypted), reducing the value of message transport 
compression techniques.  My very subjective and anecdotal 
impression is that fewer people are routinely compressing 
attachments these days, possibly because MUAs have gotten better 
at building attachments on a one-click basis that doesn't 
provide for compression at attachment time.  Similarly, I 
suspect that the amount of material that is being mailed that 
has very low information density per number of bits transmitted 
(HTML being only one example) is on the rise.   But others may 
have different experience and impressions.
(4) Operationally, the most important requirement for 
compression arises between the endpoints of a slow and/or 
expensive and/or intermittent point to point link (at SOPAC, you 
are probably very familiar with those).  For those situations, 
usually the right thing to do is to (i) set up MX records that 
force mail to end up on the top end of such a link, (ii) have 
that machine aggregate an entire data stream, presumably 
consisting of several messages, (iii) compress that stream and 
send it via some sort of "batch SMTP" or local protocol, (iv) 
decompress and disaggregate at the far end and either go back to 
SMTP or just distribute the results into a mail store, as 
appropriate.  That model compresses not only the message content 
but also the headers and envelope and, more important in many 
cases, eliminates all of the per-message command-reply 
turnarounds (which Pipelining merely reduces).   Of course, that 
approach has been in use over such links for years and years, 
starting long before the first discussions that led to MIME and 
8bit email transport.
(5) Finally, any scheme for compressing entire message bodies 
for transport purposes that doesn't also batch the messages 
themselves needs to deal with the inconveniences created by the 
incomplete separation of envelope and headers in SMTP, i.e., the 
fact that MTAs are required to insert trace ("Received:" and 
ultimately "Reply-to:") fields into message bodies without any 
knowledge of what else is going on with the message body, 
including with previously-applied "Received:" fields.  RFC 1869 
and its predecessors, and then 2821, raised the bar, but RFC 821 
does not explicitly require that message bodies have headers, or 
that those headers be in 822 format, as long as it is possible 
to prepend those trace fields.
Today, I would also worry a bit that compression might turn out 
to be the enemy of various strategies for early interception and 
repelling of spam.  One should at least think about that issue 
when contemplating compression schemes.
All of that said, if you think it would be worthwhile, 
especially after thinking about (4), I'd recommend proposing it. 
You would need to think carefully through the model and 
practical implications of (5) but, otherwise, an appropriate 
ESMTP extension wouldn't be hard to design and write up.  If you 
had such a proposal written up, even in outline, the right place 
to discuss it is probably the ietf-smtp list, hosted at imc.org. 
Only by starting from a specific writeup would you be likely to 
get a good handle on whether the idea would get enough traction 
to be worth pursuing further.
regards,
   john

p.s. Don't you know you aren't supposed to raise technical 
issues on the IETF list?  It might drop the noise to signal 
ratio below infinity, which many of those who seem to post the 
most messages to the list might find very disappointing.   :-(