Frank Ellermann wrote:
Radu Hociung wrote:
[I'm reading p* / p.* / o.* where you say o*]
None of the o* modifiers have any effect at all on SPF
results.
Strange, but I understood your later argument about sharing the
available TXT space. You're not trying to reinvent something
like SRV or NAPTR using SPF-TXT records, or are you ?
If you're talking about RFC2915, no, I don't think I'm trying that, but
I have to say, I'm not familiar with that RFC, I only just had a cursory
look at it now.
this example, let's say that the domain owner publishes an
odk=yes extension.
Anything with "yes" or "no" is a candidate for op= and all its
restrictions, simply because "odk=yes oyx=no" needs 14 char.s,
where "op=dk.noyx" needs only 10 char.s.
Correct. for DK, a "yes" or "no", binary type of information makes a
little sense, but no other info does.
I like your yx example. I understand the concerns around o*. Let's make
it o.* instead.
See also chapter 2 ("motivation").in the op-drafts -04 or -05.
perhaps only it's existence.
Tricky, it can also exist without op=dk, and that's again the
general problem of _all_ modifiers in v=spf1 introduced after
v=spf1. Same problem for spf2.0/pra and all its variations.
It's okay for a yx (not DK) where op=yx is required right from
the start. If op=yx is _required_ the domain owner must add
it to his sender policy, or he can't use yx (whatever that is).
You could limit this requirement to "if and only if you have a
v=spf1 sender policy it MUST announce op=yx". With this trick
yx could be also used independent of v=spf1.
My intention was not to create any new requirement.
I thought the o.yx would point to the prefix where that information can
be found.
If you have a small setup, your two records, spf1 and xy may very well
fit in the available TXT space, and then you don't need any pointers
from one to the other.
But if you have a big setup, such that the two records summed up would
exceed 400 bytes, then you can use the o.xy=_{anything} to offload the
top-level TXT record by moving the xy information to the prefix _{anything}.
Even the {anything} needs not be fixed. It's real location is given by
the o.xy modifier.
This is why o.xy is not there to say whether there is an xy record, but
to give an alternate location if it could not fit at the top level.
One other thing, without such an extension, future "xy"-like protocols
will be forced to implement some type of redirection, because "those SPF
{people} only leave them a few measly bytes to work with in the TXT record".
I agree that when SPF moves to a dedicated SPF RR, this need for
interoperation disappears, and it can even be made a requirement that
SPF records published on SPF RR's should not contain o.* extensions.
But if you say "yx MAY be announced as op=yx" it doesn't work.
Any MAY is implicitly "maybe not", and if the absence of op=yx
means nothing, then its presence is a waste of characters.
So op=dk doesn't fly unless the DK-spec adds a corresponding
MUST and "we" fire the still unofficial op-I-D (= SPF Council
wants it, anything else for a proper I-D would take me a few
minutes, I'd know how to fix its obsolete RfC 2234 reference.)
FAIL on their check of the MAIL-FROM. The SPF checker passes
the spf_also_found["dk"]="yes" to the MTA. In order to
decide whether to reject after the SPF fail, or to give it
one more chance using DK, it may use any other information it
has available.
That's "modifying SPF results" in my terminology. So you'd use
op=dk in a way like op=trusted, to bypass SPF in certain cases.
That makes sense - limited, because I wouldn't want to look
"into" the DATA, but reject with 5xx right after the MAIL FROM.
The result returned by SPF is the same, regardless of the contents of
the o.* extensions. But the MTA action taken may be different.
Similarly to a virus checker, its conclusion for whether a file is
infected or not is not changed by the action you wish to take (delete,
fix, quarantine)
How the various spam fighting methods are combined is the MTA
vendor's problem.
Sure. Some like Hector won't look into the DATA, it's against
their religion (shared by me in this case), others can delay
their decision after they've seen the final "dot" of SMTP DATA.
I hate looking into the DATA as well, but not as religiously, because I
hate missing legitimate email more. :)
I especially dislike being ridiculed for an over-zealous spam filter, as
it is perceived by less technically inclined relatives.
Let's assume that SPF3 operates on the message headers.
Forgetting my religion for the moment... ;-)
Thank you ;)
Let's also assume that SPF3 is designed to be a 1-query
protocol
Yes, the strange B64 or whatever it was proposed by David here.
they have to share the 400-bytes available in the TXT space
between the two SPF records.
Horrible. They better stay as far as possible away from TXT
or the future SPF RR, but okay, it's only an assumption.
Granted, it's not pretty, but SPF does use a general purpose RR... if it
were using it's SPF RR, the rules would be slightly different, but then
we would still be discussing how to get people to adopt and embrace SPF. ;)
the domain owner will double the number of queries received
from the installed base of SPF checkers.
Not really. Your assumption is actually very similar to the
situation today for v=spf1 and spf2.0/pra, e.g. ebay.com. You
already "have" both policies after one q=spf query, or at most
two queries if you need a secod q=txt. For "have" read: both
records are in your DNS cache, in the worst case.
Well, if ebay were to optimize it's SPF1 record, it would still end up
with a list of IPs so long that would require 818 bytes to represent. (2
queries at least)
I don't have an SPF2.0 compiler, but let's say that maybe their spf2.0
record could also be simplified to 818 bytes (also 2 queries, assuming
450 bytes per record).
But now adding both records to their top-level TXT, and sharing the
available space equally, they'd have do something like this:
----- ebay.com DNS answer: -----------
ebay.com. TXT "v=spf1 (200 bytes) redirect _s1.%{o}
ebay.com. TXT "spf2.0/pra (200 bytes) redirect _p1.%{o}
----- _s1 DNS answer: -----------
_s1 TXT "v=spf1 (450 bytes) redirect _s2.%{o}
----- _s2 DNS answer: -----------
_s2 TXT "v=spf1 (168 bytes) -all
----- _p1 DNS answer: -----------
_p1 TXT "v=spf1 (450 bytes) redirect _p2.%{o}
----- _p2 DNS answer: -----------
_p2 TXT "v=spf1 (168 bytes) -all
So even in this case, just by publishing the spf2.0 record, they would
go from 2 minimum queries to 3 minumum queries (50% DNS traffic
increase), generated by SPF1 checkers. Since there are no/few spf2.0
checkers, their spf2.0 record produces no ROI for the 50% increase in
DNS traffic.
Another good test case is aol. They saw this problem, and made both
their spf1 and spf2.0 small enough that both fit in a 480-byte reply
packet. For this they had to resort to using IP4 lists with /23 and /24
CIDRs, even though they probably don't use that many outgoing SMTP
servers (their spf1 policy covers an IP space of 2034 or so unique IPs).
I am guessing that based on AOL's requirement that their currently
whitelisted correspondents publish SPF records to maintain that status,
that they are planning on starting to check SPF records.
Thus, they may have given a lot of thought to performance issues
relating to SPF, and their current record is a witness to that work.
Their record requires 1 query (+1 for the PTR)
On the other hand, I don't think ebay gave much thought to checking SPF,
and implicitly not too much thought to their published SPF record, and
their sloppy, 12-query record is proof of that ignorance.
There's no need to add an op=spf2 (oops, I can't use a dot) to
the v=spf1 record, or an op=spf1 to the spf2.0/pra record, you
get them both with a single q=spf (or a second q=txt) query.
Only if they are small enough to fit *together* in a DNS response. If
they add up to more bytes than that, then you'd need to split them into
daisychained records.
1. Publish the 300 byte SPF1 record at the top domain, and
add ospf3=_s3
2. Publish the 300 byte SPF3 record at _s3.domain.com
That's a new idea, now you introduce a prefix for v=spf3 RRs.
It's also out of scope for an op=spf3, unless the v=spf3 spec.
always uses the same prefix _s3 for its records.
If you want an arbitrary prefix you could use p3=_s3.%{d} - but
an arbitrary prefix based on hints in a v=spf1 policy, that's a
major headache. There are also the known wildcard problems.
Oh, no... I'm not suggesting a fixed prefix. The fact that the pointer
specifies where the record lives is enough to fetch it with one query if
necessary.
Also, I believe that by %{d} you meant %{o}. I would suggest that
without this macro, another protocol, say libyx.so could read the
pointer from an SPF record itself without knowing any more about SPF
syntax than that it needs to look for a v=spf1 start and an
o\.yx=([^ ]*) pattern to get its needed prefix. then it would know to
append that prefix to the mailbox domain in question to get the FDQN to
the record it needs.
I am assuming that an MTA vendor would use an SPF3
implementation that are also supports SPF1
At this point I'm not more sure how many assumptions you have -
let's say "one too much". If good old v=spf1 can somehow help
to bootstrap v=spf3 it's fine, but I don't see it yet.
My mistake was to use SPF3 as an example, when I meant to use something
completely unrelated to SPF. Your yx example is much better, so I
adopted that.
if Sender ID had been successful and if it used the top level
TXT record
It's far too early to judge the success of Sender-ID, or to say
that spf2.0/mfrom etc. are a dead end. At the moment v=spf1 is
the only spec. that might fly, but v=spf1 is at least 9 months
older than any spf2.0 policy.
Ok.
the o* function would have to be built into Sender ID as well
You're talking about spf2.0, and there's no such thing as an
"o* function", neither in v=spf1 nor spf2.0. It's apparently
remotely related to op=, but I don't know any implementation
of this idea, the author would press me to send it to the IETF.
Actually two points:
1. If spf2.0 makes sense to be done before spf1, then the spf2.0
driver/library needs to provide the pointer for spf1 to use
subsequently. Otherwise, it does not need to. So this extension needs
not be an orthogonal requirement, ie, not all protocols must implement
it, but only those that have a more founded claim to the TXT space at
the top-level. So any such extension would need to be built into spf2.0
only if it would benefit a protocol that sits "lower on the ladder".
Certainly by the time spf2.0 runs, spf1 had already been run, so there's
no point for spf2.0 to provide any extension for the benefit of spf1.
2. The "other" function/functionality would be required for all
protocols that claim any space in the general purpose TXT record at the
mailbox domain. For each protocol, the same function will have a syntax
that makes sense for that protocol. Just that the associated
lib{protocol}.so would have to return an array like
{protocol}_also_found["blah"]="prefix"
Which I can't, as stated in the op= spec. only the SPF Council
can release it. Unless we'd declare the Council to be defunct.
I also think that some day soon we'll need to take a good look at this
Council business.
So far, I am also disapointed with the level of commitment, involvement
and leadership that it provides.
<sarcasm>
On the other hand, I am content with the amount of voting, motioning,
seconding and noting that is going on during the 90-minute monthly meetings.
</sarcasm>
But please let's keep this thread technical.
<http://purl.net/xyzzy/home/test/> + draft-hallambaker etc.
Interesting. I see also why it did not work.
Actually you don't see it, I've fixed the offending non-ASCII
manually in my copy. No idea why Phil never did this and sent
the fixed text again to the IETF. Okay, the silence of ASRG
wasn't very encouraging. And I'm also somewhat lost with this
text, all I remember is "so you can do bitfields with XML". ;-)
I meant that I see it because I don't believe reputation information can
be forwarded by the same party who's reputation we're trying to
establish. That model seems flawed to me.
if "op" has been in use already
AFAIK only here as a concept for yes / no aggregations, and as
a tombstone for some ideas (pra, trusted, helo, nohelo, etc.),
ready to be resurrected when necessary.
I do need to look at what op= was meant to do, as we may have some
overlap in what I'm trying to do and op=
the syntax makes it clear what is a mechanism and what is a
method.
s/method/modifier/ Yes, the syntax is clear, either "=" or
no "=" after the name, the latter must be a mechanism, it may
use a ":". But for stupid users I'd avoid any a= or mx= like
hell, there are enough "free" characters / names.
Yes, thank you for the regex. :)
All right, I agree that it doesn't matter what single character we
reserve for this function. :)
I would not use the "." as it provides no value other than a
little bit of readability.
Just an idea found in Phil's draft, how you could get your own
"modifier namespace". IMHO better than trying to squat all
names starting with a given charater, or even all o* minus op.
That's fine.
an SPF1 implementation should pass any other TXT records
found at the top level to the MTA.
It's in the DNS cache after a q=txt, isn't that good enough ?
I was hoping you'd ask that :)
The people who'd care most about this "others" optimization are those in
the high-performance MTA business. We're talking about MTAs that use
solid-state (RAM) disks for their queues instead of hard-drives because
the HDD access time is the bottleneck.
Have a look at this:
http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html
A quick summary:
The research was done by a university student, using door-stop machines
(K6-2 300MHz w/128MB 4.3GB SCSI). He did a comparison of the performance
of open-source MTAs.
He found that in one configuration postfix was capable of a thruput of
125 messages per second on his machine.
Based on his numbers using a "remote-smtp-sink" driver (which generates
no disk activity), removing the disk accesses allows a 10x increase in
thruput, respectively up to 1111 mails/s
I would guess that a for-profit company would use machines 10x as fast,
with 40x the memory, and with well-tuned commercial MTAs is able to get
at least 15 times as much thruput than this unversity student managed.
If they also use solid-state queue disks, this allows a decent MTA
configuration a throuput of about 10,000 messages per second.
This would mean that the total time available per message is about
0.1ms. Even with well done multi-threading, the time available per
message might be 1.0ms.
So in processing an email, only the DNS query part depends on
information not available on the MTA machine. Everything else is done
with information that is available in memory.
So, given that the DNS query has to go out to the local DNS cache over
the LAN, even a 1ms round trip time (300us for the query to go out,
400us for processing by the DNS cache, and 300us for the query to
return), you can see that the thruput of a well-tuned MTA can be easily
halfed by one query even to the local cache.
Granted, an MTA that manages 10,000 messages/s without any DNS activity
will be slowed down to 5,000 messages/s for the first TXT query, and to
3,333 messages/s for the second query. Assuming that on a high-volume
site, most needed information is in the cache most of the time.
So the question is:
When the per server thruput 5000 messages/s to 3333 messages/s, you need
50% more servers (3333*150% = 5000). In this case the extra query to
the cache costs 50% capital budget.
To most people it doesn't matter, because most mail servers run at light
loads.
But if we want the big players to adopt SPF, we need to provide some
efficiencies. When the big players adopt SPF, you have 3 effects:
1. SPF becomes something more than "an experiment"
2. Adoption rates for the rest of the domains will be spurred.
3. SPF overall will be more effective when more sites publish and check it.
Greetings,
Radu.