Re: Use of New Mask Mechanism

David MacQuigg wrote:
> At 04:06 PM 3/26/2005 -0500, Radu wrote:
>
> > David MacQuigg wrote:
> >
> > > At 02:45 PM 3/26/2005 -0500, Radu wrote:
> > >
> > > > David MacQuigg wrote:
> > > >
> > > > > At 01:33 PM 3/26/2005 -0500, Radu wrote:
> > > > >
> > > > > > David MacQuigg wrote:
> > > > > >
> > > > > > > At 11:53 AM 3/26/2005 -0500, Radu wrote:
> > > > > > >
> > > > > > > > Once again, the mask would not work as a mechanism (unless it 
> > > > > > > > was in include, like Frank mentioned), because each mechanism 
> > > > > > > > can return a match. the mask modifiers can return a match only 
> > > > > > > > after *all of them* have been checked against the IP. Think of a 
> > > > > > > > mask like m=65/6 m=214/6. For senders in the 214 net, your 
> > > > > > > > proposed -!ip4 mechanism would wrongfully declare the 214 
> > > > > > > > sources as "FAIL".
> > > > > > >
> > > > > > > Oops.  I thought I understood these masks, but I missed it.  OK, 
> > > > > > > so what this "mask" mechanism really says is, the IP address must 
> > > > > > > match one or another mask range AND match at least one of the 
> > > > > > > subsequent mechanisms.
> > > > > >
> > > > > > Almost :)
> > > > > >
> > > > > > Please, let's never call it a mechanism again, to avoid confusion. 
> > > > > > It really is a *modifier* !!! Actually a *set of modifiers* are 
> > > > > > only meaningful together. Individually, each mask modifier doesn't 
> > > > > > mean anything, because it doesn't tell enough to allow the checker 
> > > > > > to stop evaluating.
> > > > >
> > > > > Good point.  It's a modifier, not a mechanism.
> > > > >
> > > > > > And what it really says is: Somewhere in the included/redirected 
> > > > > > records, there are more IP mechanisms that match some of the IPs 
> > > > > > in the mask range. It's a "summary" of the remaining records, if 
> > > > > > you wish. The summary includes more IPs than the records 
> > > > > > themselves, but it serves well to tell authoritatively what IPs 
> > > > > > *aren't* in the subsequent includes. It also serves to tell you 
> > > > > > what the all at the very end of the record chain says the to do 
> > > > > > with unmatching IPs ("fail", "softfail", etc), so that you don't 
> > > > > > need to scan the whole chain to find out what the domain owner 
> > > > > > wants you to do.
> > > > > > Somehow I/we need to find a description of this that would be very 
> > > > > > clear, so that implementers of SPF checkers know what to do. It is 
> > > > > > clearly a description/language problem because you're not alone 
> > > > > > getting a grasp on its meaning.
> > > > >
> > > > > How about
> > > > >    record           = version [mask] terms *SP
> > > > >    version          = "v=spf1"
> > > > >    mask             = *( 1*SP m= ipblock )
> > > > > After processing the mask, if there is any match, proceed with the 
> > > > > terms as usual.  If there is no match, skip processing any terms 
> > > > > except 'all', and don't call for any remainder of a truncated record.
> > > >
> > > > Thats good, except the earliest the mask can be evaluated is after 
> > > > the A/MX/exists mechanisms. For instance, an exists:{i}.domain.com 
> > > > cannot be  covered by a mask. The only reason why a compiler might 
> > > > leave a/mx/exists mechanisms in the compiled record is if they 
> > > > contain macros that cannot be expanded at compile time (such as i, 
> > > > l, and possibly s).
> > > > I was planning on worrying about truncated records. I was going to 
> > > > assume that the checker cannot receive a truncated record, because 
> > > > the resolver library takes care of using TCP as necessary. In any 
> > > > case, for compiled records this should only happen if the compiler 
> > > > screwed up and created a record that is cannot fit in a UDP packet. 
> > > > I would treat that as a bug that must be fixed, as opposed to a 
> > > > condition that must be handled by the checkers.
> > > > So, since I'm assuming that the checker receives the entire record, 
> > > > the masks could be evaluated when the first include/redirect 
> > > > mechanism is encountered. This would mean that the mask does not 
> > > > need to cover IP addresses in the top record, as they are free to 
> > > > check.
> > >
> > > Now I'm confused.  If the reason for masks is *not* to avoid sending 
> > > multiple packets, and *only* to avoid processing mechanisms that 
> > > require another lookup, why do we need these lookups on the client 
> > > side?  Why can't the compiler do whatever lookups the client would 
> > > do, and make the clients job as simple as possible?
> >
> > Sorry for creating confusion.
> >
> > Say that you have a policy that compiles to 1500 bytes.
> >
> > The compiler will split it into 4 records, about 400-bytes each or so.
> >
> > example.com     IN TXT \
> >      "v=spf1 exists:{i}.{d} ip4:... redirect=_s1.{d2} m=-65/8 m=24/8"
> > _s1.example.com IN TXT "v=spf1 ip4:.... .... ....  redirect=_s2.{d2}"
> > _s2.example.com IN TXT "v=spf1 ip4:.... .... ....  redirect=_s3.{d2}"
> > _s3.example.com IN TXT "v=spf1 ip4:.... .... ....  -all"
> >
> > We want the mask to be applied after the exists:{i}.{d}. Since that
> > mechanism was in the initial query, cannot be expanded to a list of IPs
> > the mask cannot possibly apply to it.
>
> I think what you are saying is that the compiler can't get this down to 
> a simple list of IPs, because we need redirects containing macros that 
> depend on information only the client has.  So if we are to put the 
> burden of complex SPF evaluations on the server side, where it belongs, 
> seems like we have to pass all the necessary information to the server 
> in the initial query.  We already pass the domain name.  Adding the IP 
> address should not be a big burden, and it would have some other 
> benefits we discussed.
If you can find a way to do that and still keep the query cacheable, let 
me know. If it is compatible with the way DNS works currently, I'll even 
listen and pay attention. ;)
That 1 UDP packet might not seem like a lot. But currently it is 
cacheable and most of the time is not even seen on the internet. Making 
it uncacheable would be a multiple fold burden on bandwidth. That's 
exactly why caching and the TTL mechanism was invented, and now you 
suggest we give it up?
> Maybe I'm just not seeing the necessity of setups like the above 
> example.com.  I'm sure someone could come up with a scenario where it 
> would be real nice if all SPF checkers could run a Perl script embedded 
> in an SPF record, but we have to ask, is that really necessary to verify 
> a domain name?
The "..." imply a list of ip4: mechanism that is 400-bytes long. That's 
why the chaining is necessary. ebay.com has something like that. 
hotmail.com uses something similar too. When you have lots of outgoing 
servers, you need more space to list them, no?
> If we simply can't sell SPF without all these whiz-bang features, I 
> would say put it *all* on the server side.  All the client should have 
> to do is ask - "Hey <domain> is this <ip> OK?"  We dropped that idea 
> because it doesn't allow caching on the client side, but with a simple 
> PASS/FAIL response, the cost of no caching is only one UDP round trip 
> per email.  This seems like small change compared to worries about 
> runaway redirects, malicious macros, etc.
I'll humour you:

This server-side processing would not be happening on a caching server, 
correct? That would not save anything. I hope you agree.
So the only place where it might make a difference is if the evaluation 
was run on the authoritative server for the domain.
The problem with that, is that authoritative servers are designed with 
performance and reliability in mind (as opposed to caching servers, 
which care more about cost savings). As such, the auth servers *do not* 
do recursive queries, as an SPF record evaluation might be. They also do 
not do any caching. They respond to every query with a piece of data 
they already have in memory or on disk. If they don't have that piece of 
information, they return an empty response or "it doesn't exist" 
(NXDOMAIN). They never look for it anywhere else. That's why they are 
authoritative. If they don't know about it, it doesn't exist.
Now, the spfcompiler only makes sense if it is running on a master 
server. Itself the master for a zone is authoritative. The above 
authoritative servers are slaves. They take the information from the 
master server and disseminate it as if it was their own. It is the 
adminstrator of the master zone server that allows them to do so. No 
other server can respond authoritatively to queries for the zone in 
question.
So, the only place the spf compiler makes sense is on the master server, 
because ultimately, it is the only one who really knows the facts. When 
the facts change, the master informs the slaves, which do a zone 
transfer in order to update their databases. So the truth propagates 
nearly instantly from the master to the slaves, and as such the slaves 
can be seen as clones of the master, identical in every way, except for 
the 3 second delay it takes them to transfer the zone files.
You cannot run the compiler on the slaves, because they might each 
evaluate the record differently, as they are coping with different 
network conditions (such as lost packets, etc). In that case, they would 
each tell a different "truth" than each other and than the master 
server. In that case they would no longer be authoritative.
Now, having the master zone server respond to queries that require it to 
do calculations of any kind is an absolute no-no. That is because no 
matter how big the zone is (yahoo, rr, whatever), there is only one 
master. Ok, there may be a couple, but their job is not to respond to 
queries, but to 'hold the authority'. The slaves are for responding to 
queries.
So doing what you propose would require the DNS system to be turned 
upside down. The justification of SPF is just not good enough.
> How about this:  All SPF records SHOULD be compiled down to a list of 
> IPs.  If you need more than that, then do as much as you like, but give 
> the client a simple PASS or FAIL.  Most domains will then say "Here is 
> our list of IPs.  Don't ask again for X hours."  Only a few will say 
> "Our policy is so complex, you can't possibly understand it.  Send us 
> every IP you want checked."
That's exactly what the exists:{i}.domain does. It tells the domain 
about every IP it wants checked, and the server checks it. 
Unfortunately, it is extremely expensive because it's AGAU.
> I need to get back to designing ICs. :>)
Nah... you've got some great ideas and I value your contribution and 
feedback.
Just that this one isn't one of the good ideas. That is only my opinion, 
of course. But if you can prove that it's good, I'm listening.
Radu.