ietf-asrg
[Top] [All Lists]

Re: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

2006-04-03 18:03:13
On Apr 03 2006, Chris Lewis wrote:

In fact, receiver adjudication would likely _lower_ the FP rate. On
balance, I believe it's more likely that a user doesn't recognize
something he really did ask for (thru some twisty chain of branding,
lowering the FP rate) than "recognize" something he didn't ask for
(raising the FP rate).

That's your judgement talking, not the user. You're saying that you
can't let users full reign because that's like herding cats, it would
mess with your aggregate server statistics and thereby impact the
business. It's a very reasonable position from your point of view.

A guy like Church doesn't accept your point of view because he has
different priorities. I find it hard to agree with you because my
"natural" definition puts the user first. I can see that a clueless
user is well advised to let you make decisions for him, but not
someone who has used mail for a long time.


Thus, receiver adjudication has an error rate, further as I'll mention
below, likely _higher_ than the measures we're trying to test.


Only from your point of view. From the point of view of the user, the
errors you mention aren't FPs, they are one instance of FN (he doesn't
recognize something he asked for, ie that your system delivers to his
inbox) and one instance of FP (he does recognize something he didn't
ask for, ie that your system doesn't deliver).

Again, it's a matter of definition. Just one last point:

You can't measure the accuracy of something to .01% if the instrument
you're measuring it with is only 1% accurate.  Further, we'd find a 1%
FP rate on our filtering _totally_ unacceptable (that's 3000+/day). I'd
be looking for another job in another field if it was remotely that bad.

It's not difficult to set a fixed FP rate on a classifier and trade
off FN to achieve it. Every filter can achieve < 0.01% FP. In the
trivial extreme, no filtering at all achieves 0% FP and 100% FN.

What I just want to point out is that your filtering costs (ie the
estimated cost of an FP and the estimated cost of an FN) are not the
same as for an individual user. They make sense for you, no question,
but it's another potential source of valid disagreement worth laying
on the table. 

One way to route around this problem is to use ROC curves, which give a
global picture that doesn't depend on any one person's cost estimates.

-- 
Laird Breyer.

_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg

<Prev in Thread] Current Thread [Next in Thread>