ietf-asrg
[Top] [All Lists]

Re: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

2006-04-04 16:42:44
On Apr 03 2006, Chris Lewis wrote:

The user is first.  I simply recognize that a user's accuracy rate on
determining FPs is _lower_ than the automated systems are.

Again, I think it's due to a disagreement as to what spam is.  You
claim a user's accuracy is lower than an automated system's at
detecting consent-spam. I claim the user's accuracy is necessarily
higher than a third party at detecting what-I-want-now-spam. We could
go around in circles for a long time with this, but it makes more
sense to recognize that we're talking about slightly different ideas
of spam, which implicitly means your FPs aren't the same FPs I'm
talking about etc. 


You can't measure a voltage to 6 digits of accuracy if your equipment
only does three.

I agree, and I'm not proposing that. A typical statistical hypothesis
test has the option of being inconclusive. But here we're not even 
measuring the same FPs so it's moot. 

A ROC curve is a "Receiver Operating Characteristic". Imagine you have
a spam filter sytem and a panoply of subsystems and parameter values
to choose from. For each possible setting, you measure the (FP%,FN%)
pair on some data.  Then you plot these points in a square of side
length 100%. This gives a picture of the overall quality of the spam
filter as you insert or remove the subsystems and parameters.

It allows you to compare different spam filters over various operating
ranges, and gives insight into the accuracy tradeoffs you make when
you replace one subsystem with another.

This tutorial may be a little technical, but the important things are
the pictures.
http://www.cs.bris.ac.uk/~flach/ICML04tutorial/ROCtutorialPartI.pdf

This is a readable introduction with a lot of detail.
http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdf


-- 
Laird Breyer.

_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg

<Prev in Thread] Current Thread [Next in Thread>