-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Laird Breyer wrote:
Thomas is composing a reply to the paper, and I've taken note of some of
your comments for inclusion.
_Anything_ that uses "adjudicators" to determine whether an email is
spam or ham is clearly missing an important clue: spam isn't about
content, it's about _consent_. It is a behaviour (sending without
consent), NOT what is in the email.
It still needs a human adjudicator. In this case, a human who
interprets the CAN-SPAM regulations and decides whether (the legal
definition of) consent did apply.
First of all, CAN-SPAM doesn't define what is spam, and what isn't. It
simply defines a subset of spam that is legally actionable under US law.
The definition of spam doesn't depend on local ordinances.
Analysis under CAN-SPAM determinations simply isn't relevant, for
example, in our jurisdiction. Spam is spam, whether it's in Canada, or
the US or Australia.
Secondly, it simply isn't possible to determine to that level of
certainty "consent" in anything more than a tiny fraction of spam.
Spammers simply don't cooperate. Indeed, many of those who appear to
cooperate lie.
[I'll omit the story about proving that all of the "consents" Richter
used to justify email to ~700 of our users were fraudulent. I hesitate
to suggest that adjudicators would know what questions to ask, and how
to determine that they were fake. Let alone being able to do it at all
in a large or broad enough sample to be statistically relevant.]
DNSBLs that can, for example, detect compromised machines or illegal
methods being used to send the email have a better handle on intent,
than an "adjudicator" could possibly have.
Being an automated method of filtering, it depends on a set of criteria
and conditions stored in the program logic and database.
These criteria and conditions, if given in full to a human adjudicator,
could be judged by said human just as well with authority.
And these adjudicators would be able to make such judgements correctly?
Unlikely. And furthermore, even if they were not 100% accurate, ONLY
the overall result matters.
An IP HELO'ing as your server is unlikely to be judged "proof positive"
of compromise. It isn't. But testing shows that the number of
desirable emails mistakenly blocked by such a heuristic are essentially
non-existant.
We don't judge testing of drugs on whether "adjudicators" understand how
the drugs work, we judge drug efficacy on _results_ in trials.
Now on to your list, I answered it first but it's based on the above
paragraphs:
1) grey listing
2) sender/sender domain verification
3) Challenge/response
4) SPF and DKIM
5) PKI
6) CSV
7) Non-existant users
8) DCC or other distributed checksumming methodologies.
Similarly DNSBLs.
As Steve Atkins mentioned later in the thread, there's efficacy and
accuracy, two different aspects. I don't think there's anybody who would
dispute efficacy - rejecting at the source is clearly more efficient.
Efficacy has nothing to do with the success/failure of DNSBLs in a
general sense. Nor is rejecting at source a necessary consequence of
DNSBLs. We use DNSBLs, do rejections, but still get to quarantine the
whole email.
More importantly, Steve (and I) pointed out that these techniques simply
can't be tested to the level you might want to see, because they have to
be done with real mailstreams in realtime, and with, for example, grey
listing, it's simply NOT possible to judge in your terms whether the
email was spam or not. Applying the heuristic changes the sender
behavior, and you don't have the email to judge.
Similarly, with DNSBLs, we actually see spammers noticing that one
sender was blocked, and trying a chain of source IPs until one gets through.
The situation is simple: there's a variety of techniques that simply
cannot be tested to this level of academic certainty with "recipient" or
"recipient's judge" techniques, because applying them changes the
behaviour, and secondly, the recipient _cannot_ have anything to judge
it with.
For accuracy testing, any method has to involve a human adjudicator
somewhere downstream. Moreover, whatever the method being tested, all
relevant information (ie information used for that particular decision)
must be logged.
We already have the best possible scenario: the human _upstream_ who
noticed his email didn't get through, and then contacts the
administrators to get the problem fixed.
Letting the email get through in some sense (whether by quarantine or by
tagging mechanisms) simply doesn't work in the large scale. Given the
volumes that people get and the quantities they'd have to wade through,
they're simply not going to give an accurate picture - FPs will be missed.
Instead, we know that for the most part, only valid senders get notified
of rejections. They had a reason to send it in the first place. The
user at the other end doesn't know that there was something valid to
look for.
If a machine can take a decision, then a human who is given the same
information (no more no less) must be able to take a decision too. The
human has the last word, and that's the basis for accuracy testing.
This particular line of reasoning is to do with the notion that the
machine may be mistaken that the source IP is listed in the DNSBL under
evaluation. This is the least of our concerns.
This basis is approriate whether you take spam as "whatever I don't want"
or whether it's UCE under CAN-SPAM etc. In the latter case, the last word
would be interpretation of the law by humans in a court, in principle.
I could just imagine the industry being crippled in that fashion - you
can only tell how well your filters work by getting all the spammers in
court. If that's the criteria for success, that criteria will be ignored.
Now if you have a record of each transaction, and if you build a
system which submits a sample of these records to intended recipients
for verification, (and your recipients are willing to help test the
system, etc) then I think you have all the elements you need for testing.
I contend that sender notification is a more accurate and practical
metric. Which is what we do.
Still, we do have exactly what you suggest. Our quarantine - each time
someone forwards a message to themselves out of the quarantine (hence
suggestive of a FP), is recorded and sent to us for evaluation, where we
can tell what the original blocking reason was. I don't recall _ever_
seeing one about the XBL or SBL.
For example, when a message is rejected via dnsbl, your record
contains the dnsbl query and a statement of rejection. That record can
be shown to the intended recipient to be ok'd. If the recipient
disagrees with the machine, then that's a FP or FN.
For non-existant users, the decision is vacuous, ie anything the server
does is appropriate. For PKI, there is no need to ask a human to verify
the decryption with pencil and paper, obviously. Etc.
What does showing the exact reason to the end user have to do with
whether it's a FP of the dnsbl? A valid email being blocked is a FP,
period. Counting it as a FP against the DNSBL as appropriate, need not
be done by the user.
Secondly, there's no reason to ask a human or adjudicator to verify that
a rejection based on a given DNSBL is really listed in the DNSBL. Given
the volumes we're doing, errors of that nature would be immediately
catastrophically obvious.
Thirdly, even I'm not that sanguine about "non-existant users". Which is
why we simply don't count emails to non-existant users. Otherwise, I'd
include the CBL against our spamtrap - our spamtrap does something like
3-6 million emails/day. The CBL kills 80% of it, and in at least some
sense, that has no FPs by definition. The numbers are useful, but you
cannot reliably determine _real_ FPs from it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3-nr1 (Windows 2000)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQCVAwUBRDCL253FmCyJjHfhAQKaWQP/cmxbSKfCtOk5+C+0FoaMTCsXlhUL4rA9
158k+MW/SniLHK1cM8QG2ohhjgQZNBliFT+KFHh8+ptrs9S07khVvZ6Q26zg2GxS
VltQkU3KLPbSBQ0vrt4yvF5S0LZlwP67zPU6tGxnOgIVbBKh/LZ9oZ/n3RnPWu3Y
wBCR7WIydV0=
=Pzqk
-----END PGP SIGNATURE-----
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg