On Apr 01 2006, Chris Lewis wrote:
He's reporting for a spam filter responsible for handling at most a
handful of users. I have 65,000, and handle 500 times as much email per
_day_ as his (I believe) multiple day dataset. Sites handling 1000
times as much email as I do are ALSO using DNSBLs.
Right, this is a useful point against the draft.
The fact that we even have to answer to such nonsense, instead of
summarily throwing that in the trash is an insult to professionals.
But we do it anyway, because we're professionals.
Yes, and you're appreciated for it. But if the answer is easy enough
to dismiss it won't stop the next person from deeming dnsbls harmful
yet again. There's a lot of folklore and crap on the net, and that's
the environment in which opinions are formed. The ASRG doesn't have
the luxury of dismissing ideas from the (valuable) experience of its
members only.
I don't presuppose to cover vanity or hobby domains. Which is pretty
clearly what Church is talking about.
Right, this is a useful point against the draft.
The author of the draft isn't the only one who claims to have been
bitten by dnsbls, there are plenty of rants on the net, e.g.
http://paulgraham.com/spamhausblacklist.html
Paul chose to deliberately misrepresent the situation for political
ends. As did Moveon. As did Gilmore. And at least _two_ of those
listings were legitimate spam blocks. There are similarly plenty of
issues about other types of filtering goofing up.
Which doesn't help in rejecting the draft. I know I brought it up,
but it's best to stay clear of this kind of argument.
his usage patterns/user community
are unknown, his "preferred" filter is only described in terms of
unverifiable handwaving - no numbers, and is of a size that can best be
described as statistically irrelevant.
That should be enough.
Yes, I suggest backing that point up with the numbers you mentioned
at the top.
Okay, suggest how a spam/ham collection can be used to measure the
effectiveness of the following techniques that are, or can be, used in
an anti-spam solution:
I'll address a point you make later first, as it shows where we disagree:
_Anything_ that uses "adjudicators" to determine whether an email is
spam or ham is clearly missing an important clue: spam isn't about
content, it's about _consent_. It is a behaviour (sending without
consent), NOT what is in the email.
It still needs a human adjudicator. In this case, a human who
interprets the CAN-SPAM regulations and decides whether (the legal
definition of) consent did apply.
DNSBLs that can, for example, detect compromised machines or illegal
methods being used to send the email have a better handle on intent,
than an "adjudicator" could possibly have.
Being an automated method of filtering, it depends on a set of criteria
and conditions stored in the program logic and database.
These criteria and conditions, if given in full to a human adjudicator,
could be judged by said human just as well with authority.
Now on to your list, I answered it first but it's based on the above
paragraphs:
1) grey listing
2) sender/sender domain verification
3) Challenge/response
4) SPF and DKIM
5) PKI
6) CSV
7) Non-existant users
8) DCC or other distributed checksumming methodologies.
Similarly DNSBLs.
As Steve Atkins mentioned later in the thread, there's efficacy and
accuracy, two different aspects. I don't think there's anybody who would
dispute efficacy - rejecting at the source is clearly more efficient.
For accuracy testing, any method has to involve a human adjudicator
somewhere downstream. Moreover, whatever the method being tested, all
relevant information (ie information used for that particular decision)
must be logged.
If a machine can take a decision, then a human who is given the same
information (no more no less) must be able to take a decision too. The
human has the last word, and that's the basis for accuracy testing.
This basis is approriate whether you take spam as "whatever I don't want"
or whether it's UCE under CAN-SPAM etc. In the latter case, the last word
would be interpretation of the law by humans in a court, in principle.
Now if you have a record of each transaction, and if you build a
system which submits a sample of these records to intended recipients
for verification, (and your recipients are willing to help test the
system, etc) then I think you have all the elements you need for testing.
For example, when a message is rejected via dnsbl, your record
contains the dnsbl query and a statement of rejection. That record can
be shown to the intended recipient to be ok'd. If the recipient
disagrees with the machine, then that's a FP or FN.
For non-existant users, the decision is vacuous, ie anything the server
does is appropriate. For PKI, there is no need to ask a human to verify
the decryption with pencil and paper, obviously. Etc.
--
Laird Breyer.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg