From: Clifton Royston <cliftonr(_at_)lava(_dot_)net>
.... One
alternative is to always quote comparably computed FP *and* FN rates,
but it does seem that people have a hard time comparing those in the
common case where one pairwise comparison is greater and the other
smaller.
People who can't handle the two unavoidable dimensions won't be able
to handle a single number. The fact that it is hard to compare a
system that measures {10,100} with another system that measures {100,10}
cannot be swept under the rug by combining the two numbers.
...
The main merit to my mind of having one number is that it allows you
to make some meaningful comparisons of similar systems, where certain
factors are bounded or held constant in the comparison. In the case
you mention above, if the organization demands FP < 0.1%, and has to
choose between tuning a system to give them 0.02% FP and 16% FN, or
0.025% FP and 2% FN, dSpam might give them an idea which to prefer.
If you hold the FP rate constant or bounded by a number similar
to the underlying SMTP error rate, you don't need to worry about it
and have only one, easily understood and defended number.
Liudvikas Bukys suggested to me off-list that the measure could be
parameterized to reflect the users's perceived cost of FP relative to
FN. For example, dSpam[1000] might represent your organization which
values one lost mail more than 1000 extra spams, and dSpam[.01] might
reflect your user who cares about blocking spam 100 times more than
they care about losing mail. This is a very attractive idea, and
though it requires some thought, it might make the measure much more
useful.
I don't see a desirable different between saying "dSpam[1000] = 50" and
saying "FP=1/1000 and FN=1/50"
You guys might be onto something interesting for deciding how to
quantify both sorts of filtering errors, but you will lose that good
work if you melt it down into a single, simplistic puddle.
Vernon Schryver vjs(_at_)rhyolite(_dot_)com
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg