Re: Use of New Mask Mechanism

David MacQuigg wrote:
> At 02:45 PM 3/26/2005 -0500, Radu wrote:
>
> > David MacQuigg wrote:
> >
> > > At 01:33 PM 3/26/2005 -0500, Radu wrote:
> > >
> > > > David MacQuigg wrote:
> > > >
> > > > > At 11:53 AM 3/26/2005 -0500, Radu wrote:
> > > > >
> > > > > > Once again, the mask would not work as a mechanism (unless it was 
> > > > > > in include, like Frank mentioned), because each mechanism can 
> > > > > > return a match. the mask modifiers can return a match only after 
> > > > > > *all of them* have been checked against the IP. Think of a mask 
> > > > > > like m=65/6 m=214/6. For senders in the 214 net, your proposed 
> > > > > > -!ip4 mechanism would wrongfully declare the 214 sources as "FAIL".
> > > > >
> > > > >
> > > > > Oops.  I thought I understood these masks, but I missed it.  OK, so 
> > > > > what this "mask" mechanism really says is, the IP address must 
> > > > > match one or another mask range AND match at least one of the 
> > > > > subsequent mechanisms.
> > > >
> > > >
> > > > Almost :)
> > > >
> > > > Please, let's never call it a mechanism again, to avoid confusion. 
> > > > It really is a *modifier* !!! Actually a *set of modifiers* are only 
> > > > meaningful together. Individually, each mask modifier doesn't mean 
> > > > anything, because it doesn't tell enough to allow the checker to 
> > > > stop evaluating.
> > >
> > > Good point.  It's a modifier, not a mechanism.
> > >
> > > > And what it really says is: Somewhere in the included/redirected 
> > > > records, there are more IP mechanisms that match some of the IPs in 
> > > > the mask range. It's a "summary" of the remaining records, if you 
> > > > wish. The summary includes more IPs than the records themselves, but 
> > > > it serves well to tell authoritatively what IPs *aren't* in the 
> > > > subsequent includes. It also serves to tell you what the all at the 
> > > > very end of the record chain says the to do with unmatching IPs 
> > > > ("fail", "softfail", etc), so that you don't need to scan the whole 
> > > > chain to find out what the domain owner wants you to do.
> > > > Somehow I/we need to find a description of this that would be very 
> > > > clear, so that implementers of SPF checkers know what to do. It is 
> > > > clearly a description/language problem because you're not alone 
> > > > getting a grasp on its meaning.
> > >
> > > How about
> > >    record           = version [mask] terms *SP
> > >    version          = "v=spf1"
> > >    mask             = *( 1*SP m= ipblock )
> > > After processing the mask, if there is any match, proceed with the 
> > > terms as usual.  If there is no match, skip processing any terms 
> > > except 'all', and don't call for any remainder of a truncated record.
> >
> > Thats good, except the earliest the mask can be evaluated is after the 
> > A/MX/exists mechanisms. For instance, an exists:{i}.domain.com cannot 
> > be  covered by a mask. The only reason why a compiler might leave 
> > a/mx/exists mechanisms in the compiled record is if they contain 
> > macros that cannot be expanded at compile time (such as i, l, and 
> > possibly s).
> > I was planning on worrying about truncated records. I was going to 
> > assume that the checker cannot receive a truncated record, because the 
> > resolver library takes care of using TCP as necessary. In any case, 
> > for compiled records this should only happen if the compiler screwed 
> > up and created a record that is cannot fit in a UDP packet. I would 
> > treat that as a bug that must be fixed, as opposed to a condition that 
> > must be handled by the checkers.
> > So, since I'm assuming that the checker receives the entire record, 
> > the masks could be evaluated when the first include/redirect mechanism 
> > is encountered. This would mean that the mask does not need to cover 
> > IP addresses in the top record, as they are free to check.
>
> Now I'm confused.  If the reason for masks is *not* to avoid sending 
> multiple packets, and *only* to avoid processing mechanisms that require 
> another lookup, why do we need these lookups on the client side?  Why 
> can't the compiler do whatever lookups the client would do, and make the 
> clients job as simple as possible?
Sorry for creating confusion.

Say that you have a policy that compiles to 1500 bytes.

The compiler will split it into 4 records, about 400-bytes each or so.

example.com     IN TXT \
     "v=spf1 exists:{i}.{d} ip4:... redirect=_s1.{d2} m=-65/8 m=24/8"
_s1.example.com IN TXT "v=spf1 ip4:.... .... ....  redirect=_s2.{d2}"
_s2.example.com IN TXT "v=spf1 ip4:.... .... ....  redirect=_s3.{d2}"
_s3.example.com IN TXT "v=spf1 ip4:.... .... ....  -all"

We want the mask to be applied after the exists:{i}.{d}. Since that
mechanism was in the initial query, cannot be expanded to a list of IPs
the mask cannot possibly apply to it.

So we have two options:
1. If the mask is to cover the first TXT record at example.com,
   it cannot added to such a record.

2. If the mask is to exclude the top record, then it still has
   the opportunity to save us the remaining 3 redirect queries.
   But the the mask only covers mechanisms are are not visible
   because they are behind an "include" or "redirect".
   In the case of an A mechanism for instance, if the compiler
   could not convert it at compile time (because of a macro),
   the remaining redirect queries can still be avoided.


In the case of a policy that expands to 800 bytes or fewer:

example.com     IN TXT \
     "v=spf1 exists:{i}.{d} ip4:... redirect=_s1.{d2} m=-65/8 m=24/8"
_s1.example.com IN TXT "v=spf1 ip4:.... .... ....  -all"

Depending on how good a mask can be created, and whether the added bytes 
would cause the record to overflow to 3 records, it may be more 
beneficial to generate a mask-less record like this:
example.com     IN TXT "v=spf1 ip4:...            redirect=_s1.{d2}"
_s1.example.com IN TXT "v=spf1 ip4:............ exists:{i}.{d} -all"


Again, judgement on what masks are better, and what optimization 
techniques should be used are left to the compiler-designer's 
competence, as long as the meaning of the record does not change.
We are only trying to standardize how the mask is interpreted, as all 
clients must interpret the same mask identically, even though different 
compilers may come up with different optimizations of the mask and record.
In fact, the reordering of exists would change the behaviour of the 
record but not the result to the client. Ie, if the exists mechanism is 
used for tracking, and not for providing information to the checker, 
then this rearrangement would break that. Even so, when evaluated by a 
checker, both records would be equivalent.
I guess it would be best if the compiler had another command-line switch 
like -allow-tracking, which would force it to not reorder mechanisms 
with macros, but that would implicitly disable masks.
Radu.