On Apr 16, 2011, at 4:27 AM, Nathaniel Borenstein wrote:
Although all my basic instincts are in agreement with Keith's position -- the
only way you will ever persuade message senders to comply more fully with the
standards is to reject their non-compliant messages -- my recent experience
has turned me 180 degrees and convinced me that this position is hopelessly
idealistic.
As some of you know, I have only recently returned to work at a company
(Mimecast) whose primary business includes the maintenance and operation of a
large-scale MTA. The situation has changed a lot in the last ten (not to say
twenty) years, and in many ways not for the better. Today's commercial
reality is a race to the bottom that more or less guarantees that MTAs will
do precisely what Keith says they should not do, which means, to my mind,
that it behooves us to document our collective wisdom on the least harmful
and most consistent way to do it.
A single example should make this clear: Nearly every business now receives
its mail via some third party email security mechanism -- perhaps an
appliance, perhaps locally installed code, and increasingly a cloud-based
service. Competition among these third parties is fierce, and inevitably
standards compliance is far less of a driver of business decisions than
customer satisfaction. Imagine, then, that company X, which enforces the
standards very strictly, as Keith advocates, manages to "steal" a customer
from company Y, which does not. All of a sudden -- and I assure you this
really happens -- a host of slightly-misformatted but customer-desired
messages which were delivered by company Y will start getting blocked by
company X. The customer, inevitably, will scream bloody murder. They won't
care that company X is doing a better job of enforcing the standards, they
will only care that company X is -- wrongly, in their eyes -- blocking
desirable streams of messages t!
hat the previous vendor delivered with no visible problem. The generalization
is clear: the financial and business incentive is to deliver every message
that you can possibly figure out how to deliver, so that your service doesn't
appear "inferior" to customers who don't give a rat's rump about the details of
standards compliance.
This is why I think the checking for syntax and either correct or rejection of
malformed messages needs to be done on the sender's side, by the MSA.
But there's also the opposite argument - say that handling of incoming messages
moves from company X to company Y, and malformed messages that were blocked by
company X are now passed by company Y. As a result, the client company
receiving such messages, now finds itself subject to drastically increased
attacks that use malformed messages (and UA susceptibility to them) as a
vector. Now the client will scream bloody murder because Y did not block such
messages.
Both scenarios argue for more uniform handling of messages, and/or more strict
adherence to *822/MIME syntax. so there will be less variation between how
different MTAs/providers handle mail. (Though the mail screen/delivery
providers might want to distinguish themselves from one another by how
strict/tolerant/clever they are, thus increasing the variation between
providers.)
But yes, the desire for uniform handling might argue for defining a standard
syntax verifier (with some recommendations about correction, where appropriate)
and encouraging MSAs to implement it. The desire for more uniform handling of
messages might also argue for trying to move such verification/correction to
MSAs and having downstream MTAs treat such messages as opaque, so that there
will be fewer opportunities for messages to be subjected to variable amounts of
verification/correction.
(Honestly, I have long thought that traditional SMTP store-and-forward these
days is more trouble than it's worth, and what we need is to reduce the number
of hops that a message endures as it goes from sender to recipient. To this
end, I'd support some sort of pass-through extension to SMTP to allow SMTP to
continue to be used as a means to let messages get through firewalls, but
without actually doing store-and-forward when not necessary.)
I'm not sure why this came as such a surprise to me, as it is actually just
another instance of Postel's Law. Being liberal in what we accept means, in
this case, accepting and delivering any message where we believe, with a high
level of confidence, that the sender's intentions are clear despite its
standards violations.
Postel's Law was great in ARPAnet days, but I have long wondered how well it
scales to a network with billions of users and thousands or tens of thousands
of implementations.
But I don't really have any problem in principle with correcting messages where
"the sender's intentions are clear". In practice, I wonder how many cases
exist for which code can reliably detect that "the sender's intentions are
clear". For example, a missing Date header field - sure the MSA or downstream
MTA can supply one, but it doesn't really know the date/time at which the
sender caused the message to be sent.
I can tell you that nothing the IETF says is likely to have any effect on
whether or not Mimecast tries to "fix" and deliver such messages -- we do and
we will, as do nearly all of our competitors, because although we care what
the IETF says, we have no choice but to care what our paying customers say
even more. However, if the IETF offered guidance on the least harmful way to
do this, the odds are good that we would follow it. And I think it would be
better if vendors who felt the need to "fix" messages would at least be
mutually consistent in how they fix them.
One (probably controversial) idea that might reduce our collective angst
would be to specify that, when an MTA fixes such a message, it also generates
an explanatory/warning message back to the sender or the sender's postmaster
(perhaps limited to 1 message per day per malformation type). That way we
might not be exacerbating existing problems quite as much as Keith and other
(myself included) fear. And the senders of misformatted messages might
eventually fix their code, if only to shut up the annoying warning messages.
-- Nathaniel
Well, if there were a standard set of tests to apply to outgoing mail, and
these tests were implemented by MSAs, those MSAs could presumably make such
data available to sending UAs and/or their users. And yes, hopefully MUA
vendors would make sure that messages generated by their MUAs passed such
tests. That would be, IMO, the best of all possible outcomes.
Keith