On Apr 02 2006, Steve Atkins wrote:
If you reject the message during testing where you would have
accepted the message in operation then [...]
invalidates any data that includes any future delivery attempts
Yes, we agree on that. Another way of saying this is that predictive
simulation isn't truly possible because the internal state of the
spammer's software is unknowable. Thus comparing two filtering methods
by _simulation_ , ie creating hypothetical worlds where each method is
used on its own, only makes sense in some cases.
It makes sense when the simulation is valid, eg for testing local
content filtering inside a mail client. It isn't valid for dnsbl
comparisons at the SMTP level because as you say the internal state of
the spammer software is unknown.
But predictive simulation as a way of comparing two filters (or one
filter, and the absence thereof) is not the only way to do accuracy
testing. I was saying that humans who adjudicate ex post facto the
machine decision rules (rather than the messages themselves, which may
or may not be available) can validly count misclassifications after
the fact.
This gives a valid measure of accuracy for the effect of a single
filter, that can be summarized in terms of FP and FN. In turn, FP and
FN can be compared at an abstract level.
However, you are exactly right in pointing out that multiple filters
cannot be therefore compared on the same datastream when predictive
simulation isn't possible.
But even that is only part of the story. If you take two spam datastreams
and wait long enough, you'll get some identical messages in both streams.
If you prune these datastreams to keep only those identical messages, you
can compare two filters on the same data, even though they operate on
separate streams.
That's not much use in general, but suppose you're comparing two
dnsbls, and suppose these dnsbls identify spam messages by requiring
at least two identical message submissions. In that case, comparing
the intersection of two datastreams makes a lot more sense, because
those dnsbls only advise on multiply sent spams.
--
Laird Breyer.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg