Ideas for semantic (feature) extensibility

I will take a stab at identifying some requirements and methods for 
semantic extensibility.  Please comment... I would like to collect comments 
from SPF users and then include MS folks in a second round of comments if 
we all feel like we're going in the right direction.

Here are the goals I want to get to:

1. Let's assume that new keywords and new features will crop up.
2. Assume that any client might see some keyword or language element for 
which it doesn't have the feature, and in most cases doesn't understand the 
new feature.
3. Allow the publisher of the record to define the "fallback position" 
(i.e. the logic for dealing with an unknown feature.)  There are a wide 
range of different situations... we can't really solve the problem by 
mandating one and only one fallback position.
4. The publisher needs to know predictably and reliably what will happen on 
the receiving end, both with the new feature present and without.
With these in mind, rolling out a new feature should be possible, but we 
would need to add a little bit to the language.

(I am sidestepping XML vs SPF TXT for now.  Here is how. :)  Based on some 
conversations at the meeting, we think that XML allows for "syntax 
extensibility", meaning you can add new keywords to the language pretty 
easily.  It's also fairly easy to add to SPF TXT using the same concepts... 
in either case, the hard part is that you must agree on what to do when you 
come to an unknown mechanism.  The examples I'm using here look like SPF, 
but please don't take that to be an endorsement of one or the other.  For 
the record I support the proposed compromise of "SPF and XML running 
concurrently for now" and it seems likely that we will need to solve both 
problems anyway.)

The essence of the question is, "Here is this unknown/unimplemented 
feature, and I can't use it like you want me to, so I need to skip over it 
intelligently while still honoring your intentions."
Here is an example of one possible fallback situation

 mydomain.com. v=spf1 +ptr +mx +domainkeys -all

"If the message is not from my servers (by ptr or mx) it may also be 
allowed if it has the new feature "domainkeys".  Otherwise, no exceptions 
are allowed."
Here are a couple of ways to state what this publisher probably wants:


0. The current SPF strategy is:
   v=spf1 +ptr +mx ?all
   v=spf2 +ptr +mx +domainkeys -all

This allows you to do exactly what you want, but the version numbers of the 
record mean that you have to repeat yourself.  Also, some folks have given 
feedback that the version number would limit SPF to only those features 
recognized by a centralized version numbering authority.  We can sort of 
get around this by pointing out that version numbers could be like 1a or 1+ 
or similar.  But, if there is more than one optional feature released 
later, you may get an even worse situation: 1, 1d, 1j, 1dj.  Also, the 
backward compatible records will be slow to go away... and since they are 
all TXT as far as DNS is concerned, you have to return all of them, hence 
there is less space for each iteration.
So, what follows are three additional methods/mechanisms that would add 
by-feature extensibility with a predictable fallback.

1. Structured logic:
   +ptr  +mx   IF domainkeys { +domainkeys -all } ELSE { ?all }

The structured logic provides a very specific "fork" in the road... if the 
feature is available, take one track, if not, take another track.  Either 
track can have multiple follow-on steps.   This is a somewhat heavy 
solution syntax-wise, I think, because we are adding delimiters and 
nesting, as well as conditional tests.  (I am pretty sure we don't want 1. 
-- it's just here for comparison with what other higher-level languages 
might do.)

2. Choose one and only one failure action.
   +ptr  +mx  unsupported=?  +domainkeys -all

The modifier "unsupported" tells what action to be taken when an unknown 
feature is encountered.  That action will probably be one of:
   unsupported=?  Stop processing and return "unknown" ("neutral")
   unsupported=-  Stop processing and return "Fail"
   unsupported=+  Stop processing and return "Pass" (pretty unlikely ;)
   unsupported=abort  Stop processing and return error result  (current 
behavior)
   unsupported=continue  Ignore the unknown keyword and resume with the 
next step
This provides a predictable action if the feature is not supported, though 
it will only do one action, not a new sequence.  However, the record could 
include another unsupported flag later in the record, so that one feature 
fails using "unknown" and another feature fails using "Fail".

3. Trap style
   +ptr  +mx  trap(unsupported=break){ +domainkeys -all }  ?all

There is no "test" as there would be with if/then/else, but there is 
nesting, which allows you to do pretty much the same things.  The trap 
could be any one of the actions from #2, in addition to "break" which means 
"skip to the end of the trap section, then continue".

Brief discussion comparing 1, 2, 3...

My thought is that 1 is the hardest to do and 2 is the easiest (and closest 
to the current SPF spec).  3. introduces nesting, but not conditional 
tests, so it should be relatively easy to implement, but because it is 
"structured" it will give people the "illusion of complexity".
I feel that 3. is the direction that gives us the greatest number of 
options later down the line.  With a Trap{} mechanism you would be 100% 
sure of what will happen if your features are missing, and it also gives 
some more clever options that linear processing doesn't give you.  BUT, 
there are probably very few cases that would need 3. that are not solved 
adequately by 2.
If we were talking only about SPF alone, I would probably lean toward 2. to 
keep the nesting out of it and keep everything linear.  However, if we are 
thinking about XML, it already has this type of nesting built in.  Consider 
the following xml-like snippet.
 <ptr/>
 <mx/>
 <trap unsupported=break>
   <domainkeys/>
   <all action=FAIL />
 </trap>
 <all action=UNKNOWN />

So, I guess given the courtship period we are entering with MS CID, I would 
want to take a second look at option 3. (Trap mechanism) and not dismiss it 
out of hand.  However, I would love to hear feedback from others on this.

(Here is where I digress a bit from "adding features" - this example is 
about more things a Trap mechanism would allow us to do, other than feature 
extensibility)
The "trap style" could also be extended to protect you from other errors, 
or even to include/reference someone else's record in a way that you 
control.  For example, here is another scenario altogether that might 
benefit from a trap mechanism.
 mydomain.com:  +a +ptr include:comcast.net include:earthlink.net -all
 comcast.net: +a +ptr -all
 earthlink.net: +a +ptr -all

In this case, you are saying that your mail may come from your servers, 
Comcast, or Earthlink (perhaps you have users on those two networks).  If I 
understand include: correctly, if the result for comcast.net is PASS, then 
the result is also PASS for mydomain.com.
Here is what happens with include:comcast.net..
 comcast.net results:    mydomain.com results:
 PASS                    PASS
 NEUTRAL, FAIL           CONTINUE
 ERROR                   ERROR
 No SPF record exists    FAIL

I believe the redirect=comcast.net would return either Pass, Fail, or 
Unknown, depending on whatever the straight comcast.net test would have 
been, but since their record ends in -all (or any "all") then my processing 
would never continue on to Earthlink.  (Comcast could assist me by 
publishing a redirect.comcast.net that has +a +ptr +mx but no -all, I think 
if the redirect doesn't contain an "all" it is possible to come back to the 
main line.  But, still no way to change a + for them into a ? for me)
Now, let's say what I *really* want is to return "unknown" (neutral) for 
Comcast, since I have some users, but there are also a lot of forgeries 
coming from there.  If Comcast says it is a PASS, maybe I really want to 
interpret that as an unknown.  In other words, I am going to (include, 
redirect, whatever) to comcast, and I'm fishing for a + or ? which I will 
treat as a ?, and if I get a - or error I will move to the next fork.
So, what I would like in this case is something like this...

 mydomain.com: +a +ptr
     trap(+=?,?=?,-=break,error=break){ redirect=comcast.net }
     trap(+=?,?=?,-=break,error=break){ redirect=earthlink.net }
     -all

Anyway, that's a quick example of how a trap might help in other ways not 
having to do with adding features...  Kind of a "nice to have" but not a 
hard/fast requirement, I think...

--
Greg Connor <gconnor(_at_)nekodojo(_dot_)org>