spf-discuss
[Top] [All Lists]

Re: softfail considered harmful

2004-02-19 19:18:12

----- Original Message ----- 
From: "Alex van den Bogaerdt" <alex(_at_)ergens(_dot_)op(_dot_)het(_dot_)net>
To: <spf-discuss(_at_)v2(_dot_)listbox(_dot_)com>
Sent: Thursday, February 19, 2004 6:29 PM
Subject: Re: [spf-discuss] softfail considered harmful


You mean you can treat it was a "no decision" to be made, i.e, you can
not
reject.  A pass can not be trusted. More testing maybe warranted.

If the sending entity (the domain, not the user) tells you the host is
under control of said entity, you should be able to trust the message
envelope.  The return-path _is_ valid.

SPF only validates the domain part of the return path, not the user part of
the return path.

So for example, a SPF compliant spammer can issues return paths:

           knownuser(_at_)spf-spammer,com
           unknownuser(_at_)spf-spammer,com

Both pass the SPF test if the spammer is useding his machine.  However,  in
reality only the first address is valid, not the second.   In order words,
if this was the only SPF test done, the unknownuser(_at_)spf-spammer(_dot_)com 
is now
the bounce address if a bounce is inevitably determined to be required.

In short, spammers can become SPF compliant and that only validates the
machine.  It doesn't validate the complete return path address, only domain
part of the return path address.

We have an CBV system which proves all this.   Validating the machine is not
enough.  Its that simple.


Bounces are not evil.  Bounces are a necessary part of todays email.
Without SPF, you have no way of knowing (for sure) if the bounce is
directed at the true author of the message.  With an SPF-pass, you
know with a reasonable certainty the return-path is valid.

Not so, as I shown above.  The spammer can use SPF to validate the machine
and still use a bad user name as part of the complete return address.

SPF-pass:     100% certain the message is legitimate
SPF-fail:     near to 100% certain the message is bogus

No.  The opposite

SPF-fail:     100%  trust in rejection
SPF-pass:  100% certainty in the MACHINE, not the user or message.

SPF-softfail: message likely to be bogus, but far less than 100%

I agree, especially when you consider the ultimate intent of the SPF record
to be switched to a fail fallback.  But you can't make a decision, so you
need to still apply further scrutiny on the transaction.

SPF-unknown:  as if SPF were not there

same as SPF-pass, SPF-softfail, or nothing there.

A SPF fail result provided 100% trust and a 55x rejection ...
                                               ...  In this case, a
"Receive-SPF: fail" header has no role or meaning.

Not 100%, not necessarily 55x rejection, "Receive-SPF: fail" won't be
generated if the mail is rejected but must/should be generated if the
mail is _not_ rejected (for whatever reason).  SPF does not dictate one
should reject mail.

If you are not going to accept the transaction *dynamically* at the protocol
level, then the Received-SPF: will not be required to be created since there
is no message received.

SPF says in Section 3. SPF Record Evaluation

   An SPF client evaluates an SPF record and produces one of seven
   results:

    ...

     Fail (-): the message does not meet a domain's definition of
     legitimacy.  MTAs MAY reject the message using a permanent
     failure reply code.  (Code 550 is RECOMMENDED.  See RFC2821
     section 7.1.)

This is protocol level dynamic SMTP rejection. Therefore there is no
Received-SPF requirement.

The only time you need a Recieved-SPF is for systems that do not perform
"Local Recepient Validation" (such as gateway software) in which case, the
message is received and Received-SPF: will most definitely help any post AVS
validation concept.  But this contributes any plays into the bounce problem
exploited by the Sorbig-generation email viruses.

So as you see, there is no difference between softfail and pass. they
both
need to be treated the same.

Why?

Because you accepted the message in the first place at SMTP.

There _is_ a difference.  Softfail will give me, the receiver, a chance
to let you, the author, know you are doing something wrong.
(or maybe the domain owner is wrong, in which case you, the author, should
contact your provider).

The only different is a difference in "information" but has no direct
correlation to nothing you can trust!  But even then, look at what you just
done, you care creating a bounce and you are assuming the bounce address is
valid when you only checked the domain,  not the user part of the address.

Here an example of a spammer spoofing the system with a softfail:

A message from alex(_at_)aol(_dot_)com to unknown(_at_)winserver(_dot_)com

The SMPT client connects from IP 1.2.3.4 to MX at mail.winserver.com

client ip: 1.2.3.4
S: 220 mail.winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
C: HELO mail.spammer.com
S: 250 mail.winserver.com, Pleased to meet you.
C: MAIL FROM: <alex(_at_)aol(_dot_)com>

Now much of the discussion here all depends on how the SMTP server is
designed or configured to operate. Does it use a Dynamic SPF Validation
concept or POST SPF validation concept?

Lets look at both, first with a POST SPF validation concept, meaning, the
MAIL FROM is accepted:

S: 250 <alex.com>... Sender validation pending. Continue.
C: RCPT TO:  <unknown(_at_)winserver(_dot_)com>

Now much of the discussion here all depends on how the SMTP server is
designed or configured to operate to perform local user validation. Does it
use a Dynamic Local User Validation or Post Local User Validation?

Lets look at both, first with a Post Local User Validation, meaning, the
RCPT TO is accepted:

S: 250 User <unknown(_at_)winserver(_dot_)com>  ok
C: DATA
S: 354 Start mail input; end with <CRLF>.<CRLF>
... client sends data ending with final . line...
S: 250 message accepted
C: QUIT
S: 221 closing connection

Please note, Some systems will do a dynamic DATA validation, but lets not
complication this any further.  The message is accepted now.   By SMTP
standards, it is now your responsibility to either deliver it or make a
valid attempt to send a notification back to alex(_at_)aol(_dot_)com indicating 
why it
was not delivered.

Now, your new SPF post validation software will check the return path at
AOL.COM and finds the client IP address does not match, but because of the
softfail,  you don't know. You have to do more test, like AVS testing.

So you perform your post validation AVS testing and find that the message
is:

a) VIRUS - you should reject the message and NOT bounce it back. This is
what SORBIG depends on.

b) SPAM - rejection is based on your user contract/policy.  Everyone differs
here.

c) NO-SPAM - you now allow your user to decide based on the Received-SPF
header.

But now you have to validate the user and you find that it is
unknown(_at_)winserver(_dot_)com is not valid so you have now have to bounce the
message back to alex(_at_)aol(_dot_)com (except for the virus rejections).    
Systems
that validate the user first before the AVS test will fall prey of the
sorbig distribution problem.

So here you have many issues in regards to bouncing.

Lets now analyze the dynamic SPF validation system:

client ip: 1.2.3.4
S: 220 mail.winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
C: HELO mail.spammer.com
S: 250 mail.winserver.com, Pleased to meet you.
C: MAIL FROM: <alex(_at_)aol(_dot_)com>

A dynamic SPF check will behave as follows:

SPF-fail - you immediate reject the message with a 55x response per section
3 of SPF specs.

SPF-pass - you allow the session to continue with no real gain in knowledge

SPF-softfail - same as SPF-pass, you are left "unsure" of what to do, so you
allow the session to continue

SPF-none, unknown, same as pass and softfail, you allow the session to
continue.

But we talking about softfail, so lets see what happens next.

S: 250 <alex.com>... warning: SoftFail SPF result. Continue.
C: RCPT TO:  unknown(_at_)winserver(_dot_)com

The dynamic Local User validation will say:

S: 552 Sorry, User unknown.

The transaction ends, no bounce problem.

The post Local User validation will say

S: 250 User <unknown(_at_)winserver(_dot_)com>  ok

and we have the same bounce and overhead related issues as in the post SPF
validation example.

I hope this was sufficient to show the difference.

The main point being is that softfail is a LOOPHOLE into allowing spoofed
mail into the network.  Systems like GATEWAYS or relay host systems or
systems that perform post validation operations will fall prey to the
loophole.  Spammers can use return paths that point to SPF softfail systems
because it allows the non-matching IP to be accepted and since most systems
have not yet touch based with dynamic complere return path address
validation,  you fall victim to bouncing issues.

In systems that perform dynamic valdations concepts at the protocol level
benefit greatly as a system and contributes tremendous to the network
because it is not burdening the network with bounces.

Systems like ours will not even bother with SPF until the RCPT TO: is
validated.   So the user is unknown, the transaction ends right there and
then.

If the RCPT TO: user is known, i.e., hector(_at_)winserver(_dot_)com, the SPF 
softfail
result is inclusive so a final CBV check is done which AOL will fail because
alex(_at_)aol(_dot_)com is not a valid user at aol.com.   It will look like 
this:

client ip: 1.2.3.4
S: 220 mail.winserver.com Wildcat! ESMTP Server v5.7.450.9b13 ready
C: HELO mail.spammer.com
S: 250 mail.winserver.com, Pleased to meet you.
C: MAIL FROM: <alex(_at_)aol(_dot_)com>
S: 250 <alex.com>... Sender validation pending. Continue.
C: RCPT TO:  hector(_at_)winserver(_dot_)com
... the SPF, CBV testing is done
S: 552 Sorry, Return Path Not validae (Rejected by CBV)

Note, this is not a discussion about CBV and it can easily spoofed,  but
together SPF and CBV,  it has proven to a tremendous spoof-buster.  More
importantly, the overhead and bounce related issues are no longer an issue.

*THE* benefit of SPF with respect to bounces is that it limits bounces to
joe-jobbed messages.  True bounces are not affected by it, at all.

and what I am saying is softfail for a system does not perform a complete
return path validation (which currently is the majority of systems) opens an
LOOPHOLE into system.

Sure you will perform all the post validation AVS checking, spamassasin,
etc, but the biggest benefit of SPF or any concept that a SMTP system can
use at the protocol level is when it can reject the mail immediately.  Once
you accept the message,  it is your responsibility to deliver or notify the
sender why it wasn't deliverable.  It has nothing to do with mail content.
Even if it was clean or not detected as a spam or a new virus has creep in,
you have just create a bounce to a spoofed address.

All I am saying is that softfail puts a dynamic validation system in
jeopardy in the same level a post validation system is currently in.  It is
a loophole.  We have less of a problem because of the CBV, but the problem
is still there.   All I am suggesting to Meng that they makes it very clear
in the SPECS that softfail is a temporary operational mode and not
considered permanent.

Just consider the software optimization logics we are already added, as an
option for SPF processing.

If a SPF record has a softfall fallback, we don't even bother to process the
directives except for fail directives which my be used to stop a sender from
using certain machines.  But if its all pass directives, they will be
skipped when the final is softfail.  This is consider an optimization
feature to reduce DNS overhead.

We learn this the hard way.  I'm sure others will eventually see this too
when softfail becomes a norm for spammers.

Anyway, thats my take on this. :-)

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com



<Prev in Thread] Current Thread [Next in Thread>