spf-discuss
[Top] [All Lists]

Re: Re: HELO versus MAILFROM results

2005-05-06 11:16:07
Hector Santos wrote:
----- Original Message -----
From: "Radu Hociung" <radu(_dot_)spf(_at_)ohmi(_dot_)org>

I am partially guilty for the beginning of this "fud", as I said once
that anything goes in the HELO name. I'm sorry. A lot of stuff goes, but
not quite everything. I should have been more clear.

However, I still think there is no point checking the HELO.

So you would skip the obvious local domain spoofing?

No, but I would not use SPF for this purpose.

Local domain spoofing is trivial to detect, and in fact MTAs have been doing it for ages (I think).

The MTA knows which domain names it is supposed to accept mail for. It can then check helo's that are in those domains by doing an A query and comparing with the IP.

Since the zone file for those domains is either under the direct control of the MTA domain or it is trusted as being under friendly control (ie, when an MTA does backup-MX for another domain, the domain being backed-up is friendly), the A query reliably tells the MTA if the incoming connection really is within the domain or not.

You don't need SPF for this.

Also, this simple check is much less overhead than doing any SPF check. Again, this simple verification of the HELO was not invented by SPF, it has been there a long time.

SPF does not improve on this technique, IMHO.

Also, my observation is that whenever I get connections with HELO of my domain name, the following MAIL FROM is also from myself. I haven't studied this too closely to know if it is true 100% of the time, but I have never noticed a HELO of my domain followed by MAIL FROM from another domain.

So in this case, the existing local-domain check that the MTA does, is good enough to reject such connections with 1 query to the *local DNS*.

Even if the MTA did not do this check, the SPF check of the mail-from would catch the forgery.

However, the MTA is also configured to not accept MAIL-FROM as my own domain without AUTH.

The only value of SPF being involved with HELO that I can see is to hide configuration problems in the MTA (ie, why would you ever accept mail from yourself without AUTH, and why would you not have the simple HELO check that the MTA already does?)

Sendmail offers the AllowBogusHELO option. I can't find now how it does the A query on the local-domains, perhaps it doesn't, because it's not important.


All a spammer has to do is find a host name, *any host name* without a
TXT record, and use that in the HELO.


This would be a good thing:  Force the Spammer to Change!

A good thing, but not good enough... they've been changing and adapting for years. How much longer do we want to let this game continue?

This SPF check is so easy to bypass, it's not funny.

But the same applies to the MAIL FROM too.

No, it does not, due to reputation issues. Please see my message in reponse to Frank's :

http://archives.listbox.com/spf-discuss(_at_)v2(_dot_)listbox(_dot_)com/200505/0138.html

The fact that the name in the HELO is inconsequential for the rest of
the SMTP conversation means you can't tie the helo to anything else in
that email communication... so what's the point assuming it has
any meaning?


It has meaning. It must be FQDN.  Spammers are indeed learning that SMTP
compatibility is becoming more important, not less.

There is no doubt the trend is to be more correct with the conversation than
not.  Even if spoofed, the trend is to fake you out with a "emulated"
correct conversation.

The fact that it must be FQDN is a requirement, not a meaning. It's like filling red-tape forms... they're a requirement, but they don't have meaning.

IMHO, we are forcing spammers to learn SMTP compatibility forces them to learn about SMTP, and about SPF, and to be smarter about looking for the weaknesses. At the same time we're building in some weaknesses. So when the spammers catch up, we'll be back where we started.


Here's a short list of good candidates.

my-hostname-is-longer-than-yours.mit.edu
didnt.doit.wisc.edu
com.com.com
sci.fi
pearly-gates.vatican.com
diplomatic.passport.ca


And an adaptive system can quickly adjust itself.

Perhaps an example would be in order. How would a system adapt when it receives 10, or 50,000 messages a day with {random}.com.com.com as HELO and blah(_at_)yahoo(_dot_)com(_dot_) And what if the 50,000 messages each pick a different HELO from a list of 1,000,000 ?


The only check that might be remotely valid is to check the A record to
ensure it matches the IP address. But that's not SPF, and it can also be
easily faked with the [1.1.1.1] notation. Rejecting those is also not
SPF stuff.

No system worth its salt will rely on just one system.

Well, adding more systems is not a good substitute for one reliable system.

So what does SPF have to do with HELO ?


The best you can do is to protect local domains and this gives you 100%
trusted results.

Correct. But SPF is not needed to reinvent the wheel.

Actually, before it is proposed that HELO should be done, it should be
experimented with, to see how much junk it keeps away. Then, if the
results of this experiments were favourable, we could discuss including
it in a spec.

This is why I say that SPF is still at the early stages of experiment.
There are more things we haven't learnt yet than things we have.

Many have already learned all this.  This isn't today's news!

Maybe it's just that I'm not aware of all the lessons learned, or maybe I see a lot of evidence that the scenarios I consider have never been considered before.

One criticism I have of the SPF community is that it assumes the spammers are incredibly stupid. I can't equate that assumption with "learning experience".

Take the following:

char **helo_list = { // wildcard domains without SPF
 "com.com.com",
 "diplomatic.passport.ca",
 "santronics.com",
 etc...(198 others)}

char **source_list = { // mail domains with known good reputation and no SPF
 "yahoo.com",
 "gardener.com",
 etc... (10,000 others)
}
for i = 0 to i < 1,000,000 {
  printf ("HELO %s.%s", encode64(rnd(1000)), helo_list[rnd(200)]);
printf ("MAIL FROM %s(_at_)%s", encode64(rnd(1000)), source_list[rnd(10,000)]);
  ... etc
}

Unless I am mistaken, this would cause 2,000 queries for your santronics.com TXT if most of the 1,000,000 recipients checked HELO with SPF.

Regardless whether you actually have a record or not, 2,000 queries per email campaign is not something to sneeze at, if that piece of software is run 20 times a day.

The worst of it, this program would be put on a CD-ROM, to be used for years to come. Spam software from way back when is still in use today, though it yields relatively low delivery rates. By the same token, once your domain name makes it onto one of these CDs, you'll be paying your DNS service provider real dollars for years to come.

But the key question is ... what have you achieved by checking SPF that could not be achieved by checking MAIL FROM alone?


SPF evolved from the work of DMP and RMX.  DMP  had both a MFROM and a HELO
check.   DMP enforced a DNS record publishing for the email domain and the
client domain.  SPF screwed up by ignoring the HELO checked that DMP was
doing because MENG did not understand why it was needed.

I believe I understand the claims you make about usefulness, but disagree thoroughly. There is a big difference between this and not understanding why HELO checking is needed for SPF.

I can't claim I am familiar with DMP, so I cannot say if its use of HELO is well motivated or not.

Remember, the #1 protection that any LMAP concept can give you is 100% trust
in local domain protection.  The trust in protecting remote domains is less.
No one should be using YOUR domains on your server and this represents
atleast 12-15% of the transactions in the vain attempt by spammers to
emulate an inner network machine.

Isn't trust a black or white issue? I am unfamiliar with the concept of "less trust". If you cannot trust the remote domain protection, it means the remote domain protection does not exist.

For local domain protection, SPF is not needed.

Now, the WHOLE problem with LMAP in general is the DNS overhead issue.  DMP
knew that it has a DNS overhead issue with its dual lookup.   Meng tried to
minimize this by bypassing HELO altogether .  But that was a mistake.  It is
illogical.   Again, if you can' t be SPF ready at email domain without being
SPF ready at the client domain and the exception to this is
rouiting/forwarding issues.

If HELO was free from a DNS perspective, I couldn't care less either. However, since it has a cost, I need to see value in doing it. If there is no value, the cost should be eliminated (not minimized). If there is a little value, I would want to minimize the cost.

But there is no value that SPF adds to a HELO check that the simple A query for local domains already provides.

Look at this:

    Client IP:  1.2.3.4
    HELO mail.santronics.com
    MAIL FROM: address(_at_)spf-ready-domain

Even if the email domain is validated agaist the IP, this is an obvious
SPOOF because a) not only is the client domain local (I own it), but b)  it
would fail the SPF check.

There is NO possible topology that makes this a valid transaction.  We know
it because its our domain, thus 100% trust.
But what if its another domain?

Yes, the trust is lower.   What the SPF specs is suggesting is that you have
the option to perform an SPF check.

I think we are mostly agreeing, except that while I consider "lower trust" to be equivalent to "no trust", you are more lenient, and allow an unknown trust level to mean somewhat trusted. Do you have a grade of "untrusted" in your logic? How do you assign the "untrusted" grade to HELO?

This is how I see for the above example when the client domain is remote
(not local, we don't own it).

Given  SPF(MFROM) = PASS

    SPF(HELO) -> NXDOMAIN -->  overall result ->  FAIL  (High Trust)
    SPF(HELO) -> FAIL -->  overall result ->  FAIL  (High Trust)
    SPF(HELO) -> PASS -->  overall result ->  PASS  (High Trust)
    SPF(HELO) -> NONE -->  overall result ->  PASS (Lower Trust)

This doesn't make sense...

How can the the MAIL FROM give PASS and the HELO give FAIL?

If the sender's policy is +all, which gives the MAIL FROM a PASS, who are you to say that the sender intended FAIL ?

This has all been discussed Radu.  SPF must take HELO into consideration and
allow implementations to work with it.   You can't exclude it.     Even if
the SPAMMERS adjust, that would be a great thing because we are making them
do things in OUR direction.

If it has been discussed, and agreed upon, why is it so hard to prove the checking HELO is reliable and useful?

Are there any research numbers showing the success rate being higher with checking HELO than without checking HELO? If SPF were a mature experiment, these numbers would exist somewhere.

I will admit ignorance when I see research being done. Until then, I only see opinions, and while I respect everyone's right to have opinions, the opinions themselves have little value if they're not substantiated with any shred of proof.

It is frustrating being the only one contributing time to do this kind of research. The other members of this list must have logfiles too. I see no shortage of opinions, but very few facts.

No offense intended at anyone's address.

Regards,
Radu.