spf-discuss
[Top] [All Lists]

Re: Re: DNS load research

2005-03-21 20:04:12
On Mon, 2005-03-21 at 17:47, Radu Hociung wrote:
Andy Bakun wrote:
While, yes, you would have to look up the MX for domains that you would
not normally correspond with when resolving an SPF record that uses an
mx mechanism, this is no different than having to handle increased email
load in general.  The result of which is largely cachable.

Correct me if I'm wrong, but I thought we're fighting spam 

_I_ thought we were fighting forgery.  The "increased email load in
general" I was referring to is load that goes up over time as you get
more customers, make more contacts, communicate with more people.  The
difference between hotmail.com the day it was opened to the public and
the hotmail.com of today.

because we 
want to decrease the amount of resources wasted as a result of it. 

I don't think this is possible short of everyone being honest and there
not being any bad guys.  This is a naive position to take, which is how
we got into this forgery mess in the first place.  SPF is a social
contract; I tell you my sending policy, you enforce it for me. 
Enforcement doesn't come free.  If we want to enforce the rules
(previously, there were spoken rules, but they were abused, so now it
comes to having to enforce them by rejecting mail), does that mean that
the resources it takes to do the enforcement were wasted?  Yes, we want
to avoid throwing the baby out with the omelet, but you have to break a
few eggs if you want to make bathwater.  Or something like that. ;)

SPF 
only exists because of that desire, so why does it not make sense that 
the traffic that it itself generates is as low as possibee?

It does make sense that the traffic should be as low as possible.  But
at the cost of the other benefits?  Especially when low traffic can be
achieved through other means (like SPF aware DNS servers).


The good point of simplicity and convenience is about the only reason 
why a _little_ extra traffic can be tolerated. I think this may be a 
good argument, but I subscribe to the idea within reasonable limits.

As someone else said, both the sender and recipient will see the harmful
effects of increased load.  That should be enough of a reason for anyone
to simply their records and/or their infrastructure.  But removing the
possibility of making the hard things possible (by lowering the limits
or removing mechanisms) should not be one of the goals.


As I've said before, the mx mechanism eases maintenance, and if we need
to encourage use of SPF compilers or add code to DNS servers to replace
mx with a list of ip4s at query time, then so be it.  Using arguments
like "you can do easy maintenance with a makefile and macros" is
disingenuous because not all DNS information, even within the same zone,
is maintained by the same person with the same permissions on even the
same system.  There are LDAP and RDBM systems out there that store
enterprise-wide network topology information, and different departments
can be responsible for different records and parts of the tree that
eventually get served by DNS.  Guy recently said it better than I am, I
think.

I don't know how that works, but I'll look into it. Any info you can 
point me to would be appreciated.

I'm not sure exactly what you are looking for pointers on, but there is
http://cr.yp.to/djbdns/other.html


As for the thing I said I'll get to in a minute... looking up MX records
returns the A records of the values of MX in the additional section of
the result, which makes both aol.com and leave-it-to-grace.com MX look
ups only 1 query.  

This was the meat of this paragraph, BTW.  This is an optimization that
can happen immediately (depending on when people update their DNS
software, of course) if DNS servers become aware of the SPF format and
include those other records it knows about in the additional section of
the response.


Again, there's nothing to stop a smart DNS server
from populating the additional section of the result of a TXT query for
an SPF record with information to avoid further queries.  If this
existed (and the market might bring this about) the exact cost (in
number of queries) is then different based on the exact version of the
DNS software being run by the SPF serving domain...

This is very interesting, it would make sense if aol.com used custom 
changes to their DNS servers to optimize their load, but do you know if 
other standard servers include this kind of functionality ?

None.  I assert this is because of lack of market pressure to produce
those features.  It doesn't exist (yet) because when we talked about how
SPF should be deployed, making changes to DNS servers was deemed "to
difficult to get deployed".  Now, things are different.  If people can
automagically reduce the PHYSICAL number of queries by upgrading their
DNS software, this much more likely to be deployed, because only those
with expensive records need to do it.


How much of the internet can do this kind of optimization currently?

Zero.  As far as I know, the market had not yet demanded that SPF record
data be delivered more efficiently by the DNS server (you're the only
one pushing for it, and you're not a market :) ).  I figure this is
because it's too early to tell.  But a good place to start would to
actually write the code that goes into packages like bind and djbdns to
solve this problem if you perceive it to be a problem.


If the goal is to have everyone publish SPF, we must deal with that 
scenario. How much will it cost our DNS infrastructure if everyone 
published SPF ? Currently only a small percentage do, and looking at the 
traffic numbers I don't like where they are headed.

The only thing I can suggest to combat this is that SPF not be deployed
at all.  Then our DNS numbers would be great, but our stats for forged
email that no one wants would be in the toilet.  I, for one, would like
to throw money at getting more bandwidth and CPU power so I don't have
to deal with forged email (_your_ mileage may vary).  SPF actually has
the capability to do this, in a predictable, scalable way that gets to
the root of the problem (unlike things like baysian filters and MUA
checks (SenderID) that require the mail to be accepted first).

"Boy, this world-wide-web thing is great, but now my DNS servers are
serving X times as many requests than when I just ran a gopher server."

Andy, I consider you to be one of the more thoughtful participants in 
this mailing list. 

You flatter me.  I bet you only think this because I spell check my
messages. :)

Please don't loose it now. :)

My point with that hypothetical quote was:

      * we need to distribute a certain kind of information
      * this information usually already exists in DNS
      * many of us consider the utility of SPF outweighing additional
        load on DNS to distribute this information

I see absolutely no reason why the DNS system should not be
exploited/used to its fullest potential in this regard.  See how far it
will stretch!  If SPF makes DNS fall down, is it SPF's fault or is it
that DNS is not robust enough to support the applications that are using
it?  While no excuse, SPF's perhaps bastardized use of DNS is nothing
compared to how HTTP has been bastardized (heh, "any sufficiently
advanced protocol is indistinguishable from HTTP", "a protocol has not
been fully realized until there is an apache module that serves via it")

It is said that SenderID abuses SPF records for a purpose they are not
intended to be used for.  It can not be said that SPF abuses DNS,
because SPF is using DNS for the exact purpose it was designed for: an 
efficient distributed method of information describing a domain.

(The fact that SPF (ab)uses the TXT record is another issue entirely and
is beyond the scope of DNS load issues).


The HTTP protocol is wonderfully light on the DNS system, probably as 
light as the gopher protocol used to be.

Yes, the "HTTP protocol" is, but in terms of the number of times that
gethostbyname and friends are called (which, short of caching in either
the client, libc or local DNS servers, would be a direct measurement of
how many queries it takes), web browsing is much more expensive than
gopher except for the plainest of pages, what with all the extra media
that needs to be transfered.

The DNS that represents a web server is only queried once for each ISP 
of a visitor to the site. So if 1000 customers of earthlink.com surfs to 
www.coolsite.com, the DNS server of earthlink.com will only query the 
coolsite.com domain ONCE and then serve the response to its customers 
from its cache. For heavily used sites, there is for the most part one 
query per day per ISP (assuming TTL=1day). Looking at my own server, 
which is used very litely, I have on average 10 queries/day for 
www.ohmi.org, but about 100 web-page hits. And this is DNS if you only 
count hits, not bytes transfer, this ratio of 10/100 is very high 
probably, because I have very little traffic, perhaps 10 unique visitors 
daily who subscribe to different ISPs in the world.

How, other than the exists mechanism, does this paragraph not apply
completely to SPF?  If 1000 customers of earthlink.com send email to
their friends at hotmail.com, it doesn't matter how complex their
records are (short of using exists) because they will be cached.

I postulate that DNS usage patterns (as in "previously seen addresses
are significantly more likely than addresses never seen before") for
email are the same or less than that for HTTP/Web browsing.  There is a
high correlation between where I send and receive email today and where
I send and receive email tomorrow.  My web browsing habits are to the
same set of web sites everyday, but it is much more likely that I'm
going to off to a domain I've never seen before (and will likely never
go to again) when web browsing than when I send and receive email. 

The result of SPF should be LESS forgery.  Once there is less forgery
and domain reputation systems will be useful, there is less chance of
seeing a domain you've never seen before sending you email.  At that
point, you may even be able to reject mail BEFORE even checking the SPF
record.


As far as I can tell, the SPF system is the application with the highest 
DNS load that the DNS system has seen so far. Is there another that 
compares well to SPF?

Not that I'm aware of, but how many people are like you and do DNS load
statistics gathering?  I recently had a DNS server have a hardware
failure.  Now I'm having trouble deciding how much money to spend on a
replacement -- I don't want to spend more money than I have to, but
getting a full server seems like overkill since the machine sits at a
load of zero.  This is for a company that does Internet retail with a
staff of about 50 and sees about 600,000 page views/day.  We are not
large by any significant measure, and presumably the smaller domains
will be the ones most influenced by complex records (whether they are on
the sending or receiving end).  I do not believe that deploying SPF will
make a measurable impact on our DNS load nor on our bandwidth usage, and
if it does, those are things I can easily throw money at to "fix" (at
least on my end) if I need a complex record.

I am not comparing current volumes of DNS traffic, because SPF is just 
getting started. I am comparing the designs of the applications that use 
DNS. I think SPF is by design much heavier on DNS than anything else 
we've seen so far. When/if everyone adopts it, it will be the heaviest 
by volume too.

Even though your research indicates that a 95%+ of the domains publish
simplistic records?

Section 2 of RFC3467 is interesting.  I suppose I can only hope that the
author's remarks are more timeless than you suspect SPF will make them.


-- 
Andy Bakun <spf(_at_)leave-it-to-grace(_dot_)com>


<Prev in Thread] Current Thread [Next in Thread>