Wayne,
Thanks for posting this information.
I would like to also show some SPF stats. Maybe the DNS experts here can
make sense of any of this. My concerns are about DNS overhead and
redundancy.
As I mentioned in one previous messages about statistics, I pointed out what
seems to be real factor is not the total SPF records in DNS currently
published by how effective that would be on network topology basis or on per
node (system) basis. I used the term personal community to reflect how each
system has an unique relationship or association with its environment.
The easiest example would be a system that gets a lot of AOL.COM senders,
will see a higher SPF rate because AOL has a SPF policy published. On the
other hand, a system that gets a lot of HOTMAIL.COM senders will see a
higher MCEP rate because HOTMAIL.COM has a MCEP policy published. This
suggests that it will help tremendously to get the larger more abused ISPs
involved in protecting their domains.
It also suggest that for the domains you deal most with, you should
encourage supporting MARID.
Another example is a semi-private system, private in terms that much of the
association is established. A company, a service bureau, and ISP or even
personal system with your friends, your personal community of people you
exchange mail with, etc They are all still open and expose to the network
so SPAM is an still a major issue.
I would like to show a SPF statistic report breakdown for the month of
May/2004:
Statistics: May, 2004 day 1 to 31
total connections : 152416
total spf lookup : 4747 ( 3.1%) See Note 1.
total spf none : 4258
total spf records : 489 (10.3%) See Note 2:
total breakdown : 489 unique: 94
- pass : 349 unique: 33
- fail : 85 unique: 20
- neutral : 40 unique: 29
- softfail : 14 unique: 11
- unknown : 1 unique: 1
- none : 4258 unique: 866 (See Note 2)
Note 1:
First, we did everything we could to optimize or minimal the need to perform
spam checking. Combined with a suite of filtering methods, by the time it
reaches the LMAP lookups, only about 3.1% of the total connects are
evaluated for SPF. I would say that is an tremendous overhead reduction
improvement.
Note 2:
Of the 4747 SPF lookups, 10.3% have SPF records. That's good I guess for
SPF!. However, only 94 of them or 19.2% are unique domains. Another view
would be 5 transactions per SPF domain.
For the none results, the near same rate 20.3% appears. Most of the none
result are redundant.
Of course, these rates will differ per system. But it suggest that each
system will have a unique and high degree of association with its
environment. SPF can gain another Million nodes next month, and I will have
the same rate or effectiveness because very little of the new million SPF
domains have nothing to do with us or very little of its spoofed domains
come our way. What is important is the common systems that call us.
What does this say?
I believed it says that most of the benefit will come with each system first
protected its own domains and then the associations around it. Sounds
pretty obvious? But consider that when a virus such as SORBIG exploits an
end user, the first group of systems the virus will target is the user's
personal community, his ISP and everyone around the ISP. We seen this a
number of our ISP customers, especially older ones who were still using
non-local user validation at SMTP and depended on the gateway to bounce the
mail. They simply were not aware of the new features (new to them) such as
SMTP Local User Validation that eliminates bounce needs for RCPT level
rejections. So when their users were exploited, their server was overcome
with bounces all over the place, further perpetuating the virus
distribution.
I guess, to use a stupid analogy, it doesn't quite help calling the Roach
Exterminator to fume your apartment if your neighbors are not fumed as well.
I would love to hear some input. Maybe I am analyzing this wrong, but it
kind of tells me there will be high degree DNS lookup overhead as MARID gets
started but in the end, it will evolved to a high level of DNS lookup
redundancy.
Here is the June breakdown up to this point.
Statistics: June, 2004 day 1 to 24
total connections : 128942
total spf lookup : 1901 (1.5%)
total spf none : 1615
total spf records : 286 (15.04%)
total : 286 unique: 72
- pass : 212 unique: 25
- fail : 38 unique: 13
- neutral : 22 unique: 21
- softfail : 11 unique: 10
- unknown : 3 unique: 3
- none : 1615 unique: 649
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com
----- Original Message -----
From: "wayne" <wayne(_at_)midwestcs(_dot_)com>
To: "IETF MARID WG" <ietf-mxcomp(_at_)imc(_dot_)org>
Sent: Thursday, June 24, 2004 3:39 AM
Subject: Re: Some stats on TXT usage in domain names (updated)
About a month ago, I posted some stats on the usage of TXT records.
Here are some updates
There was a net increase of 2040 domains with SPF records, a 32.3%
increase in the last month. SPF records are now the most common type
of TXT record usage, surpassing the "unix epoch timestamp" that is
found in many zones.
The SPF adoption roll, despite being down much of this month, had a
net increase of 4776 records, a 33.7% increase. (The adoption roll,
like most SPF related things, is run by a volunteer.)
For Caller-ID records, the was a net increase of 5 records, a 6%
increase.
1289260 total domain names (same as last time)
35115 total TXT records found (some domains have more than one)
8360 have SPF records -> 23.8% of all domain level TXT records are
spf records.
26359 domains have txt records
6315 domains have spf records
18968 spf_domains adoption roll
87 have Caller-ID records
Last months stats as a reference:
In <x4wu35ml3v(_dot_)fsf(_at_)footbone(_dot_)midwestcs(_dot_)com> wayne
<wayne(_at_)midwestcs(_dot_)com>
writes:
1289260 total domain names
33016 total TXT records found (some domains have more than one)
6320 have SPF records -> 19.1% of all domain level TXT records are
spf records
26359 domains have txt records
6315 domains have spf records
1456 domains found that were also in the adopt roll -> 23.0%
14192 spf_domains adoption roll
61554 estimated domains have SPF records. (This probably misses
most parked domains, of which I have heard rumors that there
are a least a couple hundred thousand with SPF records.
82 have Caller-ID records
57 have both Caller-ID records and SPF records
45 have "testing=true" in the C-ID records.
25 have C-ID records only, of which three (10%) are microsoft.com,
exchange.microsoft.com and hotmail.com