ietf-822
[Top] [All Lists]

Re: Comments on Malformed Message BCP draft

2011-04-15 15:00:05
If it were to very carefully define exactly what kinds of checks and 
corrections were permissible and which were not, I agree.

Keith

On Apr 15, 2011, at 3:01 PM, Murray S. Kucherawy wrote:

An effort such as this one could actually improve the possibility of avoiding 
such false positives as you are describing.
 
From: Keith Moore [mailto:moore(_at_)network-heretics(_dot_)com] 
Sent: Friday, April 15, 2011 11:42 AM
To: Murray S. Kucherawy
Cc: ietf-822(_at_)imc(_dot_)org
Subject: Re: Comments on Malformed Message BCP draft
 
I've seen a lot more harm done by spam filters, than by spam.  
 
of course that's subjective, as I don't directly see the "good" that the 
filters do - when they drop spam - but I do see the harm that the filters do 
- when they drop messages that I needed to see.   but spam filters do a lot 
of harm to the reliability of message delivery.
 
Keith
 
On Apr 15, 2011, at 2:12 PM, Murray S. Kucherawy wrote:


It’s not, but it is in a better position to reject content than an MUA is 
because it’s the thing that talks SMTP across the ADMD boundary.  So where a 
particular recommendation might be “reject” or “discard” or “quarantine”, 
those are MTA actions and not MUA actions.
 
It’s also where ingress filters (spam, virus, other content) tend to be 
attached, and this document addresses those as well.  Every module involved 
needs to act in concert.
 
From: Keith Moore [mailto:moore(_at_)network-heretics(_dot_)com] 
Sent: Friday, April 15, 2011 11:06 AM
To: Murray S. Kucherawy
Cc: ietf-822(_at_)imc(_dot_)org
Subject: Re: Comments on Malformed Message BCP draft
 
It's an attack vector no matter who is interpreting the mail.   Why is the 
MTA in a better position to examine content than the MUA?  
 
Keith
 
On Apr 15, 2011, at 2:04 PM, Murray S. Kucherawy wrote:



Attempts to interpret malformed mail are what MUAs are doing, and what MTAs 
are not doing (or are doing differently).  As Dave Cridland points out, this 
creates an attack vector.
 
I didn’t suggest that the IETF needs to recommend “practically… what 
happens”, but I think we’re burying our collective head in the sand if we 
maintain the position that malformed mail shouldn’t be allowed in the first 
place.  Decades of permissive software deployment got us to where we are, and 
that’s not going to be undone in short order.
 
I totally agree, for example, that pressure should be applied to senders.  
But that’s simply not what happens.  And in that stalemate between the way we 
say it should work and the way it does work, the end user loses.
 
From: Keith Moore [mailto:moore(_at_)network-heretics(_dot_)com] 
Sent: Friday, April 15, 2011 10:46 AM
To: Murray S. Kucherawy
Cc: ietf-822(_at_)imc(_dot_)org
Subject: Re: Comments on Malformed Message BCP draft
 
IETF's job is not to recommend "practically .... what happens" but to 
recommend what works well from a technical perspective.      Masking the 
problem does not work well.    And in my experience, attempts to interpret 
malformed messages and guess what they really meant often fail.
 
However, it is important to arrange things so that the "pressure" goes to the 
right place - i.e. someone who can fix the actual problem.   
 
In the case of a malformed message, that ends up being the sender.  The 
sender might then put pressure on his ISP ("why didn't my mail get there?  
why did something complain about my mail reader generating a malformed 
message?")   And the ISP might well respond by having their mail submission 
servers try to repair those messages.    At least in that case, if the ISP's 
mail submission servers do more harm than good, the feedback will still go 
back to the sender and/or his ISP, and the ISP will be in a position to fix 
the problem with the fix.  When the "repair" is done by a party unrelated to 
the sender, that opportunity is lost, and there is no convergence toward a 
working system.
 
In general, OPES rules would seem to apply here.
 
Keith
 
 
On Apr 15, 2011, at 11:54 AM, Murray S. Kucherawy wrote:




I think that’s true from the IETF side of things, but practically speaking 
that’s rarely what happens.  What arrives at an ingress MTA contains all 
kinds of insane things, and in my experience the choice to reject them 
outright puts huge pressure on customer service facilities that nearly always 
results in demands for more relaxed software. And since that turns it into a 
business case, the pressure usually wins.
 
So given that reality, this work seems to make sense to have out there, 
rather than allowing widely varied choices about how to handle this case or 
that one that result in weak ingress security all around.
 
From: Keith Moore [mailto:moore(_at_)network-heretics(_dot_)com] 
Sent: Friday, April 15, 2011 5:51 AM
To: Murray S. Kucherawy
Cc: ietf-822(_at_)imc(_dot_)org
Subject: Re: Comments on Malformed Message BCP draft
 
I'm strongly opposed to MTAs "fixing" malformed messages (other than 
submission servers fixing a small number of known problems caused by broken 
mail clients).
If an MTA does anything at all when it thinks that a message is malformed, it 
should be to bounce it _exactly as it received it originally_.
 
MTAs trying to fix malformed messages, at best, mask problems further 
upstream that should be fixed.   At worst, they exacerbate existing problems 
and make such problems harder to diagnose.
 
Keith
 
On Apr 14, 2011, at 3:07 PM, Murray S. Kucherawy wrote:





This is some work starting up in the APPS area.  Please comment on the 
apps-discuss list if you’re interested in participating.
 
From: apps-discuss-bounces(_at_)ietf(_dot_)org 
[mailto:apps-discuss-bounces(_at_)ietf(_dot_)org] On Behalf Of Simon Tyler
Sent: Thursday, April 14, 2011 2:59 AM
To: apps-discuss(_at_)ietf(_dot_)org
Subject: [apps-discuss] Comments on Malformed Message BCP draft
 
Hi,
 
Having read the Malformed Message BCP draft I am interested in getting some 
discussion going on its content and to find the best way forward.
 
For those who missed it, the draft is at:

https://www1.tools.ietf.org/html/draft-kucherawy-mta-malformed-00

I have a few comments on it.

One thing that concerns me is the proposal that processing should stop when 
certain malformations are identified.

For example it is proposed that should a badly wrapped header field be found 
(quite how we define this is left open, I presume a line that does not start 
with a valid header field name followed by a colon) then the processing agent 
should treat this as the end of the header. My feeling is that this opens up 
a greater potential hole than the one closed and that can be exploited.

An example of the type of issue this could is cause is that should such a 
malformation occur before the MIME header fields in the header then this 
would cause the rest of the header and the message body to be treated as 
plain text. This could cause content analysis system to fail as they would 
not interpret the MIME content in the way that was presumably intended.

Given that these recommendations are unlikely to be followed by all clients 
and servers, I feel that this suggestion will allow content through an agent 
without suitable processing. My preference on this particular malformation 
would be to continue processing the header fields, this is based on the 
assumption that what follows the malformed header field is more likely to be 
further header fields and not body content. What we do with the malformed 
header field I am less certain about. We could just ignore it or we could 
treat it as part of the previous header field - both will be right as often 
as they wrong. I would welcome some additional thoughts on this.

I have similar feelings about some of the other suggestions including the 
lack of a MIME-Version header. We cannot ignore intended meaning just because 
a non-compliant application made a small error in the header fields. Everyone 
will be best served if we subject such messages to more analysis, not less.

On the whole I think a set of guidelines in this area is good but it will be 
hard to reach consensus without agreement on some basic underlying 
principles.  I would suggest that one such principle is to try to do what the 
intended recipient would most likely prefer, which is generally to fix and 
deliver non-spam messages.

I would also propose some additions to the draft. At Mimecast we see a lot of 
messages generated by all sorts of systems and amongst these we see a lot of 
different kinds of message malformations.

I'll suggest more as I think of them but for starters here are a few:

1. Excessively long lines in both headers and body. I commonly see lines that 
are several hundred Kbs in length. These are often valid messages in the 
sense that the content is desired by the receiver and in all respects other 
than line length are well formed. Obviously a limit has to be enforced and I 
would like to find a consensus on what sort of limit is reasonable. Initially 
I felt 8K was a good limit - it is after all 8 times the limit in RFC 5321. 
But it appears that this is too small a limit in real situations. When the 
limit is exceeded, what is the best option – a rejection or  forced line 
wrap. I am open to both. 

2. Invalid characters in headers. I often see headers with un-encoded 8bit 
characters. These are often displayed correctly to the recipient where the 
client happens upon the correct character set by virtue of chance.
 
3. 8bit characters in MIME sections with a content-transfer-encoding of 7bit.

If you have read this far then I think you will agree with me that Murray has 
made a good start on a much needed set of guidelines. Now let's see if we can 
support him to expand on the work he has done and reach a consensus on the 
best approaches.

Simon
<ATT00001..txt>