On Apr 03 2006, Seth Breidbart wrote:
Consider some spamware: based on whether it gets 4XX or 5XX, early or
late in the transaction, it decides how to continue (4XX resends the
same thing later, 5XX early sends from another IP address, 5XX late
changes hashbusters and sends from the same IP address, a second fail
changes hashbusters again and sends from a different IP address, where
to send other spams going to the same mailserver but to different
addresses from, . . .)
Now, whatever you do affects what you see later. Therefore you
_cannot_ do a test without perturbing the system.
I don't think we're talking about the same thing. When I'm talking
about measuring, I'm not talking about intervening directly, making a
choice about 4XX or 5XX on the spot.
First, your greylisting system does what it normally does. The spammer
reacts and life goes on. A week later (say), you go through the logs
and you verify that the system is not losing mail you think it should
accept. Think of it as debugging, QA, whatever. That's the accuracy
test.
You summarize this QA with a FP/FN rate, which counts the number of
mistakes/bugs of each type. There you are, an FP/FN rate. You can now
compare these numbers with FP/FNs obtained from another spamfilter on
comparable data.
--
Laird Breyer.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg