I was among the early participants constantly recommending on doing delay
verification and the idea of Final Destination vs. Routed mail. So I'm
happy to see some of my input was considered and added to the drafts.
This did not come immediately to mind. It came about from field testing with
hundreds of field test sites. The conclusion was obvious.
In July/August 2003, we began exploring and field testing using normal,
sequence validation of the 2821 entities using RBL and CBV method with a
SMTP 2821.MAILFROM hook we call WCSAP (Wildcat! Sender Authentication
Protocol). While CBV did a wonderful job, there was a lot of overhead
related issues to consider. This is when I started to hang around the IETF
to explore other methods being proposal. The LMAP method sounded
interesting so I started with DMP and then added SPF when AOL began to
support it.
By November, wcSAP did the following in this specific order:
Internal White/Black Rule Based filtering
RBL
DMP, SPF or MCEP (Microsoft Caller ID)
CBV
100% configurable by the SysAdmin.
We were still not happy with the overall operation and did not want to
release the product until more research was done. During the month of
December, we began to record detail statistics across all field testers
which I will summarize as follows:
- The average session time was increase to 40-60 seconds due to DNS delays
- an average of 35% of the mail was rejected by RCPT TO.
So it was quickly apparent that the number #1 problem is the tremendous
amount of LOOKUP overhead in DNS for NXDOMAIN and if not NXDOMAIN, not
compliant LMAP systems.
We explored the delayed validation so that at a minimum, an average 35%
reduction was realized in all the DNS overhead.
On December 25, during some idle Christmas hours, I added the logic to delay
the validation and it was an instant improvement:
- The average session time was decrease to 15-20 seconds
- The DNS overhead was reduced tremendously
Not related to SPF, but we also added a Multiple Line system policy display
for the SMTP Welcome and this knocked out a significant 40% of the
connections which are attributes to BULK spammers with dumb SMTP scripts
that can't handle standard multiple response lines.
In any case, we were astonished by the results and released the product to
our customers with great reviews.
In a widely deployed environment, systems will be over exerted with a
tremendous amount of unnecessary and wasted overhead in DNS lookups and
validations methods. This can not be ignored, and I strongly believe
implementations will experience the same issues we found early on.
Here are the facts:
1) Most systems effected with SPAM attacks will experience a
connections/transactions rate where the majority is problem mail
(spam/spoofed, etc). The industry figure for this varies between 60-80% of
the mail is problematic.
2) The percentage varies but we see an average of 35% of illegal or no
access RCPT TO addresses.
3) There is no solid conclusion on HOW spammers will adopt, but they will
fall under 3 categories:
a) Non-adopters, status quo.
b) Malicious Adopters who try to exploit the system,
c) Legitimate Adopters, Compliant with current laws or standards (CANSPAM,
SPF, etc).
The percentage of the categories is not quite clear yet.
What this means is simple: Most of your mail is problematic, Most of your
DNS request will FAIL (NXDOMAIN) and/or provide no results, and you still
need to deal with a vast degree of compliancy issues that have nothing to do
with SPF.
The spam issue is one that is directly related to Anonymous Final
Destination Transactions.
Routed transactions are already handled. It requires standard SMTP
validation methods already in place such as IP allow tables, SMTP AUTH and
POP BEFORE SMTP.
It is only when the RCPT TO is targeting a local final destination where
SMTP fails - this is the LOOP hole because SMTP says no validation is
necessary for final destination mail.
Therefore, I see it as a prudent technical requirement in today's anti-spam
environment that the 2821.RCPTTO forwarding address is investigated first
before any other technologies with a very high potential overhead is put on
the local system and across the network.
Sincerely,
Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office
----- Original Message -----
From: "william(at)elan.net" <william(_at_)elan(_dot_)net>
To: <spf-discuss(_at_)v2(_dot_)listbox(_dot_)com>
Sent: Thursday, September 30, 2004 2:58 AM
Subject: Re: [spf-discuss] Unified SPF Algorithm (was: moving on from MARID)
On Thu, 30 Sep 2004, Hector Santos wrote:
Is it all possible to add SPF 2.0 provision for consideration of the
most
important entity of the 2821 transaction? The forwarding address;
2821.RCPTTO?
Can you explain what you have in mind and how the checking of that address
should be done that is not being done right now?
If RCPTTO fails, then all your steps is all for nothing.
Well, if you don't know how to route the email in the first place why
bother doing any from address checking...
This is already somewhat implied by existing algorithm, although may not
be perfectly clear. In draft-ietf-marid-mailfrom it says:
Software SHOULD perform this [SPF] authorization check during the
processing of the SMTP transaction that injects the mail. This
allows errors to be returned directly to the injecting server by
way of SMTP replies. Software can perform the check as early as
the MAIL command, though it may be easier to delay the check to
some later stage of the transaction.
A modified version of that is in leibzon-resposible-submitter:
If SMTP Client provided SUBMITTER argument to MAIL command then SMTP
Client SHOULD perform authorization check during the time of SMTP
transaction. The check can be performed directly at the time of MAIL
command or it maybe easier to delay the check to later stage of SMTP
transmission.
[Note to self: change "maybe" to "may be" before publication]
Both basicly are supposed to mean that you can do SPF check directly at
MAIL command but may want to delay it until you see RCPT TO and decide
if its appropriate based on user policies. If I understand, you believe
its better instead of recomendation to directly specify "SHOULD do
verification after RCPT TO". I personally think what we have is fine
and implementors will understand this and that we should not directly
say "SHOULD". But perhaps extra paragraph why they may want to do
verification after RCPT TO would not hurt - so do you want to help and
write something up for inclusion in future drafts?
Actually some modifications to what I wrote are necessary to make
sure if PTR is not there SUBMIT would override MFROM. It should be like:
Come to think of it, the algorithm has some more errors...
Here is another try at it (probably not the last try either):
--------------------------------------------------------------------------
Step | SPF Identity | Result of verification
------+---------------+---------------------------------------------------
1 | SPF2.0/HELO | If Fail -> reject, otherwise go to 2
2 | SPF2.0/SUBMIT | If Fail -> reject, otherwise go to 3
3 | SPF2.0/MFROM | If Pass and #1 is Pass -> accept email, otherwise
4
4 | SPF2.0/PTR | If Fail and #3 is Fail -> reject, otherwise 5
5 | n/a | If #1, #2, #4 are None and #3 is Fail -> reject
| | othewise accept email
--------------------------------------------------------------------------
---
William Leibzon, Elan Networks:
mailto: william(_at_)elan(_dot_)net
Anti-Spam and Email Security Research Worksite:
http://www.elan.net/~william/emailsecurity/
-------
Sender Policy Framework: http://spf.pobox.com/
Archives at http://archives.listbox.com/spf-discuss/current/
http://www.InboxEvent.com/?s=d --- Inbox Event Nov 17-19 in Atlanta
features SPF and Sender ID.
To unsubscribe, change your address, or temporarily deactivate your
subscription,
please go to
http://v2.listbox.com/member/?listname=spf-discuss(_at_)v2(_dot_)listbox(_dot_)com