ietf-mxcomp
[Top] [All Lists]

Re: PRA algorithm and use of non-standard header fields

2004-07-20 08:47:29



On the same day that Caller-ID was announce to this mailing list
(April 6th, about three and a half months ago), I posted the following
question:

: In <9156B81DAA29204989AD43E88688FAAB01373697(_at_)df-lassie(_dot_)dogfood> 
"Harry Katz" <hkatz(_at_)exchange(_dot_)microsoft(_dot_)com> writes:
: 
: > [snip] But let's start with the base case - one address on the From line
: > - and work our way from there. 
: 
: I am very concerned with, as you say, the many special cases of
: dealing with the RFC2822 data.  While working code is not required at
: this stage of RFC development, it is certainly very helpful in trying
: to get a grasp on the whole problem and learning about subtle, but
: critical details.
: 
: Do you have any working code that validates the RFC2822 that we can
: look at, test, analyze and collect data with?  I realize that the C-ID
: doc on the microsoft website has an algorithm outline, but that would
: require a lot of work to turn into working code.
: 
: If you can't provide working code, can you provide data and your own
: analysis of the situation?



I would like to thank both Andy and Mark for finally providing some
data on the Caller-ID algorithm (aka PRA/PRD).  I think it is
especially nice of them since neither of them are the ones that have
been pushing for the PRA the hardest.


This working group is supposed to do engineering ("application of
scientific and mathematical principles to practical ends"), and it is
hard for me to believe that a huge amount of engineering has gone on
WRT the PRA when there has been no to base things on.




In <85725BC7-D99E-11D8-BD4A-000A95B3BA44(_at_)hxr(_dot_)us> Andrew Newton 
<andy(_at_)hxr(_dot_)us> writes:

I reran my tests using PRA and SPF-classic.  Here are the results:

HAM - 15678 messages (most from mailing lists, btw)
====   Msgs w/usable RR Deny  Pass
       ---------------- ----- -----
SPF-C               32% 3.9%  23.5%
PRA                 32% 4.5%  23.7%

Andy:

I'm not sure what these numbers are supposed to be.  What exactly are
"messages with usable RRs"?  Are you saying that about one third of
all email that you checked had an SPF record for the domain?  If so,
that is higher than the data I'm seeing.

The "deny" percentages are also much higher than I would expect.  Have
you checked into why these messages are being rejected?  Are they due
to forgeries, or badly constructed SPF records, or fundamental flaws
in SPF-classic/SenderID?





In <1EE98B0E-D9CB-11D8-896F-000393A56BB6(_at_)glyphic(_dot_)com> Mark Lentczner 
<markl(_at_)glyphic(_dot_)com> writes:

I took messages, computed the PRA and compared it's domain to the
domain of the envelope from.  [snip]  The
percentage where they differ is:

      ham1 17%
      ham2 16%
      spam1 8%
      spam2 9%

[mongo snip!]

1) Over 80% of mail is not complicated and the PRA produces the same
domain as env. from.  Oddly enough, even more so for spam.  (I suppose
spammers just haven't caught on.)

My guess is that the higher rate of 2821.FROM differing from the PRA
is caused by email sent through mailing lists.  I think it would be
interesting to dig into this to make sure we know what is going on.


2) While still only a small percentage of sites have records, more
sites identified by PRA had records than env. from.  This could be
because the PRA is more often a bigger site.

Again, I suspect that this has to do with mailing lists.  From
skimming my own mail folders, it appears that domains that host
mailing lists are much less likely to publish SPF records.


4) For ham, env. from produces a greater percentage of passes than
PRA.  It also produces few fails, but the numbers here are very low.
For ham, more passes and fewer fails is good.

Yes, this is a very small test case, but both your data and Andy's
seems to show that the PRA has a higher rate of rejecting valid email
than SPF-classic.



I think more interesting numbers are actually in the attachment that
Mark included that had more detailed data.

::: Data set Ham 1 :::

PRA Query Stats:            published SPF
----------------          ---------------
  pass                          11 ( 24%)
  fail                           0 (  0%)
  softfail                       0 (  0%)
  neutral                       34 ( 75%)
  unknown                        0 (  0%)
  error                          0 (  0%)

Env. From  Query Stats:     published SPF
-----------------------   ---------------
  pass                          18 (100%)
  fail                           0 (  0%)
  softfail                       0 (  0%)
  neutral                        0 (  0%)
  unknown                        0 (  0%)
  error                          0 (  0%)


Both Andy and Mark appeared to concentrate more on the pass/fail
results, but I think we need to look much closer at the neutrals.
There are quite a few major email domains that are currently
publishing SPF records that end in ?all.  I think we all want to see
the day when most people publish -all, so we need to look at why the
neutrals are happening and what can be done to make them pass.


My guess is, again, mailing lists.  If joe-sixpack(_at_)aol(_dot_)com sends 
email
to some-mailinglist(_at_)groups(_dot_)yahoo(_dot_)com and it is received by 
Andy or
Mark, the result will be a neutral for the PRA but for the 2821.FROM,
there will be no SPF record found.  This is because AOL publishes an
SPF record that ends in ?all, and Yahoo Groups doesn't add a Sender:
header to their mailing lists, although the do correctly use a Yahoo
domain for the 2821.FROM.

So, in this first set of ham, there are no outright rejections of
valid email for either PRA or SPF-classic, the PRA looks like it might
be generating a huge number of incorrect results.




::: Data set Ham 2 :::

PRA Query Stats:            published SPF
----------------          ---------------
  pass                          14 ( 53%)
  fail                           7 ( 26%)
  softfail                       0 (  0%)
  neutral                        3 ( 11%)
  unknown                        2 (  7%)
  error                          0 (  0%)

Env. From  Query Stats:     published SPF
-----------------------   ---------------
  pass                          16 ( 66%)
  fail                           0 (  0%)
  softfail                       0 (  0%)
  neutral                        1 (  4%)
  unknown                        7 ( 29%)
  error                          0 (  0%)


Again, the PRA has returned more neutrals and, percentage wise, a huge
number of fails.


As both Andy and Mark pointed out, these are small datasets and are
probably not very diverse.  It is far too little data to draw any
conclusions from, but it does show that much more data needs to be
looked at.


-wayne