This is a rough outline of the convergence between SPF and Caller-ID
that Wayne, Greg, Mark, and Meng (from SPF), and Harry, Bob, and Jim
(from Microsoft) worked out during this week of MAAWG and MARID
meetings.
It is still in flux so don't take it as a fully-baked proposal; but it
does lay out some new directions that people in the MARID interim
meeting seem reasonably happy with in principle. More than anything I
think they're relieved that we finally have some kind of convergence
so the uncertainty goes away. It's kind of like how the stock market
moves in November: uncertainty is always a depressing factor.
Doing both 2822 and 2821 is a big step forward too, and makes a lot of
the ultimate "buyers" of this proposal happy.
When you read this, pretend you've forgetten everything you know about
SPF and Caller-ID, and apply beginner's mind. People who are totally
new to all this might actually be able to understand it more easily
than people who already have some attachment to SPF.
The New SPF
Thu May 20 14:51:44 EDT 2004
CURRENT ISSUES
* Forwarding is perceived as the biggest problem with SPF.
SRS has been proposed as the workaround.
* SRS sucks. Nobody's happy with it, and it's a pain in the butt.
If we could find a better workaround that would be great.
* The Microsoft guys (and lots of other guys) really want to do 2822
verification because that's what the user sees and that's a good
contribution to fighting phishing. Nobody opposes this, except
maybe Yahoo who think we can just go straight to crypto signing
without stopping at the Designated Sender step along the way.
* The SPF guys really want to do 2821 verification because you don't
spend the bandwidth of DATA. They also want to be able to get fewer
joe-job bounces, and if everybody out there implements SPF checking
then that will come true.
* Seth Goodman and others in the SES camp have pointed out that
instead of requiring everyone in the world to implement SPF
checking, you can stop joe-jobs more unilaterally simply by
implementing SES, Signed Envelope Sender. Dave Crocker has
independently invented this also.
* Critics of Signed Envelope Sender have said that it's unrealistic to
expect big ISPs and enterprises to actually do the signing thing for
all outgoing mail.
* On the other hand, Yahoo seems to think that putting a signature in
the headers of all outgoing mail isn't so bad.
* Also, mailing lists do VERP today, and SRS and SES are pretty close.
* Finally, Microsoft has plans to do something very similar to SES,
except it's a signed header that they expect to get back in bounce
messages. So it's the same idea basically just in a different place
with different operational tradeoffs.
* So the large-scale deployability of SES is an open question we
probably shouldn't try to address right now.
THE BIG NEW IDEA: let's define a new ESMTP field, "RFROM".
* To recap the Caller-ID concept, you pull out a Purported Responsible
Domain (PRD) from the headers using some kind of selection
algorithm, and perform Designated Sender checks against that. You
may have to show users the whole trust path if there's a big
forwarding relationship.
* We expect senders to put that PRD address into the RFROM extension.
Suppose original(_at_)sender(_dot_)com sends mail to
alias(_at_)pobox(_dot_)com which
forwards to <final(_at_)recipient(_dot_)com>. When the final recipient gets
the message, they see
MAIL FROM:<original(_at_)sender(_dot_)com>
RFROM:<alias(_at_)pobox(_dot_)com>
Resent-Sender: <alias(_at_)pobox(_dot_)com>
From: <original(_at_)sender(_dot_)com>
To: <final(_at_)recipient(_dot_)com>
Observe that the Resent-Sender, which would be the output of the PRD
algorithm, matches the RFROM.
So now the SPF check runs against ... what, the RFROM?
Yes. In the Old SPF, checks ran against the MAIL FROM. This caused
problems for forwarders.
In the New SPF, if a MAIL RFROM is present, checks now run against
it.
But what if a MAIL RFROM is not present?
If a MAIL RFROM is not present, checks run against the PRD from the
headers ...
... Until the Flag Day, when you can go back to checking MAIL FROM
if the RFROM is not present.
More about Flag Days at the end.
Wait, do senders have to run the Caller-ID algorithm to extract a
PRD?
Probably not. For Caller-ID, senders are prepending stuff to the
headers. They just need to take what they prepended and dump it
into RFROM. They don't have to *read* the headers, just know what
they're prepending.
So there are a few corollaries:
* If the RFROM turns out to be different from the PRD in 2822, reject
the message with confidence. So even after a receiver does an SPF
test against the RFROM, they still need to run the Caller-ID
algorithm against the 2822, just to confirm that the PRD from the
2822 actually is the RFROM from the 2821.
* If the MARID lookup on the RFROM address is anything but a PASS, you
can safely reject it. The justification is: if you're making up
RFROM, you know what you're doing. This prevents spammers from
forging innocent third party domains into RFROM, which is a huge win
over the old SPF, where spammers could forge innocent domains who
don't publish SPF into MAIL FROM.
Earlier you talked about what forwarders should do. Do you have
prescriptions for all the different kinds of senders?
Yes.
SPF asks certain constituencies to change (eg forwarders have to do
SRS).
Caller-ID asks certain constituencies to change (often to prepend a
new header.)
The short answer is, the New SPF asks the above constituencies to do
both, except instead of SRS, it's RFROM.
The long answer is, we'll have full prescriptions by Monday.
SEMANTIC CONVERGENCE
* The SPF model is procedural: start here, try that, if that matches,
short circuit.
* The Caller-ID model is set-theoretic: define the good, define the
bad, let's locate the client in that space.
* Bob Atkinson decided that the procedural and set-theoretic models
are isomorphic. The SPF ordering is fine and survives translation
into a set model.
* Microsoft approve of all of the SPF mechanisms, including ptr and
exists.
* SPF has to add a caveat telling people that mechanisms should not
expect guaranteed side-effects. In other words, if you want to use
"exists" for logging purposes, you shouldn't; if folks do this, they
should know that their mileage may vary.
* After discussion of macros, Bob concluded that macro expressions are
sufficiently cacheable and are therefore acceptable.
* To make the above statement true, SPF has to give up the %h macro,
which it's willing to do since nobody's using it anyway.
* Semantic convergence has been achieved between SPF and Caller-ID.
SYNTACTIC CONVERGENCE
* To XML Believers, there is no extensible syntax but XML.
Surprisingly, though Meng is not a member of this religion, when he
wanted to object to this, he suddenly felt like a teenage punk
telling his elders that they don't know what it's like, man. Doing
XML just feels like listening to the advice of the elders.
* But it doesn't have to be all-or-nothing.
* The SPF syntax already goes a long way. One day we may find a need
to express concepts that are not expressible in simple SPF, and that
require XML. For example, perhaps some senders need to express
accreditation assertions in a very particular way. When that day
comes, they want to be able to switch to XML with minimal pain so
they can take advantage of those new features.
* Until that day comes, though, the simple SPF syntax will do just
fine for a huge majority of senders who don't need the extra
features.
* Observe that HTTP/1.0 and HTTP/1.1 are logically different, yet
servers can understand both, and old browsers are still welcome to
use HTTP/1.0.
* Similarly, the New SPF does not obsolete either the Old SPF or
Caller-ID. Imagine a "one interpreter, two parsers" world: we
expect client implementations to be able to parse both the Old SPF
records, and the equivalent-but-more-extensible XML representation
of same.
* Meng expects that the vast majority of folks will publish using
v=spf1 notation.
* Microsoft expects that when certain folks start needing new features
that are not expressible in v=spf1, they can publish their records
in XML and all the clients out there will be able to read those
records.
* We'll need to be very firm that client implementations be able to
read both forms.
* This is probably the most contentious issue.
* The desire for XML arises from foresight: we want to save the world
today, and we think we can do that with the existing SPF TXT syntax;
but we also want to make it easier to save the world in the future,
with less work than we needed this time round. And that is very
challenging because we don't know today what we'll be saving the
world from tomorrow. So it seems wise to let XML take care of
extensibility, at least at the syntactic level. (We'll need to
define semantic extensibility no matter what we go with, XML or
something else.) Also, Meng had the nagging feeling that if we
didn't allow an upgrade path to XML, one day someone would say to
him "told you so", and he hates that.
* The desire to support the "legacy" SPF syntax arises from the
principle: keep things simple for the simple cases; and to avoid
burning the early adopters. I would be quite happy to keep telling
people to publish in SPF syntax if they don't need any of the
extensibility points and new features you find only in XML.
* The desire to support both syntaxes arises from "be liberal in what
you receive".
THE NEED FOR A FLAG DAY
* The promise made to the SPF camp was: you will one day be able to
reject forged MAIL FROM based on SPF lookups. I intend to keep that
promise.
* Under the Old SPF, until the forwarders were all SRS compliant, we
used a hacky workaround with trusted-forwarder.org.
* Under the New SPF, until the forwarders all use RFROM, we are in the
same boat.
* Either way we need the industry to announce a flag day: we need to
strike a balance between giving forwarders lots of notice, and being
able to block stuff before DATA.
* Changing the burden to RFROM instead of SRS makes things a lot easier
for forwarders. That reduces their adoption threshhold. Everybody
wins.
* We can't decide today when the flag day will be, but one day soon we
probably will be able to.
* Before the Flag Day:
- Campaign for MTAs to start supporting RFROM.
- In the absence of RFROM, receivers SHOULD NOT reject spoofs until
after seeing data, and SHOULD use responsible addr from 2822
header.
- (Receivers who know exactly who's forwarding to them may choose to
reject anyway. Local policy, etc.)
- Fight Joe Jobs with SES.
* After the Flag Day:
- MAIL FROM RFROM now expected from forwarders.
- MAIL FROM spoofs without RFROM MAY be rejected.
- Flag day requires industry coordination to ensure MAIL FROM /
RFROM support is out there.
- RFROM harder to spoof because only a PASS result is acceptable.
BUT WHAT ABOUT PREVENTING JOE-JOBS?
* Under the Old SPF, we wanted all the SMTP receivers in the world to
become SPF aware, and to reject mail that failed SPF tests.
* Hosts that do not upgrade will continue to send bad bounces. This
is seen as a moral failing on their part.
* Under the New SPF, senders pragmatically and unilaterally implement
SES. They get the benefit of joe-job protection without waiting for
the whole world to change.
* Now, doing SES may be impractical for big ISPs. But SES and
worldwide adoption of SPF can meet somewhere in the middle.