----- Original Message -----
From: <owner-ietf-smtp(_at_)mail(_dot_)imc(_dot_)org>
To: <winserver(_dot_)support(_at_)winserver(_dot_)com>
Sent: Tuesday, March 08, 2005 5:28 PM
On 3/8/2005 11:36 AM, Hector Santos wrote:
Over the pass year and a half, we have proved that by increasing the
level of SMTP compliancy required by senders, you can address an
extremely high rejection rate with a very low to non-existence false
positives. 100% based on SMTP compliancy.
I'm seeing just the opposite. I've been doing some expirementation with
some strict SMTP rules in SpamAssassin (looking for things like literal
HELO, mismatched rDNS, etc), and am getting reports of extremely high
error rates from testers. [1]
You are not going to get a reliable result performing DNS techniques for
HELO/EHLO. Thats a given.
I don't see how you can get poor results with testing domain literals.
Either it is correct or its not.
Our HELO tester is quite simple:
1) Test for 2822 ABNF compliance. - RFC 2822, RFC 2821
2) Domain Literals MUST match the IP - RFC 2821
3) Check for Local Domain Spoofs - RFC 2821
Examples:
1) No Brackets
HELO 208.131.247.9
Technically incorrect. No brackets, before anyone stops here and says "Ah
ha! Got you!" read below.
2) IP mismatch
CLIENT IP = 205.123.343.23
HELO [208.131.247.9]
This is a clear violation The IPs much match! Again, read below.
3) Local Domain Spoof
CLIENT IP = 205.123.343.23
HELO mail.winserver.com
There is a clear local domain spoof. As a compromise to having an opening
ended DNS lookup, we only check for local domain specifications. If that
domain is part of your domain ownership, it can't be used by some foriegn IP
address and sender.
These are all lightweight testing with an extremely high rate of
effectiveness. I don't see how you can get a high false positive here. If
you do DNS lookups, sure. That isn't going to work well.
We have over 1.5 year of production statistics that proves all this:
http://www.winserver.com/spamstats
Another high rejection feature is having a multiline welcome response, this
will knock out atleast 40% of your streaming bulk spammers who are using
broken small footprint mail senders with little logic in it to test for
multiline line response codes. I've been waiting for these idiots to
adjust - it hasn't happen yet!
If you want to catch the Major Abusers and take them to court for abusing
your system, this is how you do it because these softwares are so stupid
that they continue to re-retry until one way or another they finally learn
they are not getting in.
We got two instances of false positive reports:
1) A poor NAT configuration where the LAN IP was used for the WAN IP.
2) Old legacy software using unbracketed domain literals.
In both cases, they were quickly resolved with configuration corrections on
their end.
In a few cases with customers who still had an "not sure" feeling about this
when they stopped getting large amount of spam, turned it off but quickly
turned it back on when they saw the high effectiveness of it. They quickly
became believers.
SMTP Compliancy does indeed works and like I said, we should not be suprise.
Over 60-80% of all malicious transactions fall into a non-SMTP compliancy
category. Therefore, without a question, applying pressure at SMTP will no
doubt be extremely effective.
Note that I implemented these tests in SA so that I could encourage people
to STOP doing hard-rejects tests at the MTA level, due to very high
false-positive rates. I thought that by assigning a probability (as
opposed to a simple pass-fail test) that these tests would be more
effective, but from what I've seen so far, there are just too many
mis-managed mail servers out there to even establish a probability rate
that's acceptable.
There are, of course, some very useful pass-fail MTA tests (including
things like non-existent domain names), but not nearly as many as I
suspected going into this.
[1] http://www.rulesemporium.com/forums/showthread.php?s=&threadid=105
I will check out your information, but I am not into playing "probabilities
and statistics" when it comes to SMTP. Either the client is SMTP compliant
or its not. There is no in-between and it is 100% completely independent of
the mail content considerations (the payload).
Our wcSMTP system implements two hooks:
1) RCPT state transaction validation hook (WCSMTP)
2) DATA state Mail Content Hook (wcSMTPFILTER)
The WCSMTPFILTER is a skeleton. We are not in the business of interpreting
mail content. Administrator define the mail content filtering policies.
For SMTP, we just want a compliant transaction. Administrators augment
3rd party AVS solution with wcSMTPFILTER which may include wsSpamGuard,
SpamAssasin, McAfee, Norton, or some Sieve hook, etc All WCSMTP wants when
it calls the wcSMTPFILTER blackbox is three possible answers using a 32 bit
high/low response value:
- TRUE - accept, 250 response fixed
- FALSE - reject high word has response (default 451 if I remember)
- KEEP - accept and put aside. 250 response fixed.
We really don't want to add any payload analysis baloney into the SMTP
process. It would mean a new product liability issue. If it was 100%
perfect, no problem, it would be considered. But if not, we leave this for
the admins to define and setup. That has been my long time technical
opinion for all mail networks since the dawn of time. Rejection at the
transaction level is more plausible Acceptance of the mail for post process
analysis assumes a responsibility for delivery or notification and this has
been exploited by the SORBIG-generation e-viruses.
So the faster you reject before the PAYLOAD is received, the better the
customer is served.
All our old UUCP/SLIP custom who updated to SMTP, fell into this category.
They didn't want to even bother with all the new SMTP level validation
features, especially RCPT Validation at SMTP. This was normally done at the
gateway. So an older ISP calls and says "I am getting murdered" I tell
them to simply turn on RCPT validation at SMTP and SORBIG is practically
eliminated from propagating thru his system. Turn on Sender validation
stuff, and his malicious transactions are practically 100% eliminated. Of
course, valid transaction spam does make it in, and that is where SA and
others are needed.
Look, all my point is here is that we really have no choice by address
malicious mail by fixing the very issues they exploit at SMTP. Since it is
unfeasible to believe SMTP will change in the foreseeable future, at a
minimum, promote a stronger enforcement of SMTP client/server compliancy
using the current standard.
I believe once this is done across all major vendors, the industry will
benefit. Only at this point, we can then invent new ideas that work around
a SMTP compliant transaction expectation. We can't just continue with
stupid kludged ideas that is based on non-compliant operations.
Sincerely,
Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office