On Apr 02 2006, Steve Atkins wrote:
Chris Lewis writes:
Justin Mason wrote:
First, you accept every message, and record the "real-time" data
points regarding how the message was listed against those services,
and/or how it *would* have been rejected at SMTP transaction time
(if
at all). However, you don't reject, you accept everything.
Not rejecting the mail alters the behavior of the sender. That skews
the data from this approach. Doesn't mean it's not a useful metric, but
it's not a real measurement, and can't be used to give good estimates
of real-world performance (at least, not without quite a lot of
additional
data, anyway).
During the SMTP transaction, the server might reject the data from the
client, and would do so on _some_ grounds (eg IP address). Strictly
speaking, _these_ grounds are what needs to be logged and verified by
a human somewhere down the line.
As such, it's not necessary to accept the message at all and no sender
skewing takes place. In a formal test, a human adjudicator may well
decide that a single logged IP address + dnsbl response and without an
accepted message body are insufficient grounds for summary rejection,
and count it as an FP. That's still a valid test though, because the
human observer takes the final responsibility, and can explain his
reasoning if necessary.
For example, the criterion might be consent (as in CAN-SPAM) and the
adjudicator might be a lawyer, and a single IP listed on a dnsbl might
not constitute proof of consent or dissent under the regulations.
And to measure actual efficacy you need to actually reject mail,
if what you're measuring is behaviour if you reject mail. (If the
behaviour you're trying to model is "what happens if we bulk folder
or devnull mail detected by a filter?" then that's not a problem,
but one of the usually mentioned advantages of DNSBLs is that
you don't need to accept the mail).
True, but in that case it is doubly important to not amalgamate
accuracy testing with efficacy testing. These aspects should be
clearly separated in arguments intended for non-experts.
--
Laird Breyer.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg