On Fri, 29 Aug 2003 14:23:52 -0400, asrg-request(_at_)ietf(_dot_)org wrote:
Two things on my mind:
First off, I'd like to thank Scott Nelson for his recent posting
regarding greylisting on spamtrap addresses. His results underscore
two important points that permate all of the group's
analysis-and-characterization work:
1) Although he wasn't trying to demonstrate a
reduction in spam *attempts*, his results
serve as a pointed reminder of just how
important a control condition is. That is,
if he *had* been trying to demonstrate a
reduction in spam attempts, the control
condition ("do nothing," aka spam1)
actually "outperformed" the experimental
condition (aka spam2) by a measurable
(though not statistically significant)
margin.
2) His data, which used a 3-week basis for
for measurement, were volatile enough to
preclude statistically significant
findings. (Again, not that he was
looking for them, but object lessons
never hurt.)
The second thing on my mind regards the role of a control condition
in experimental design. I wonder if I've done an effective job of
articulating just how important it is to make sure that the *only*
systematic difference between (an) experimental group(s) and a
control group is/are the independent variable(s). I'm thinking here
of the posting that suggested that control group addresses should
open spam, click-thru, etc.
*Exactly the opposite is true*. A control group is just like a
"sugar pill" condition in a drug study. The "job" of a control group
is to show the effects of doing *nothing*. Now, if one is interested
in testing the effects of click-thru, it's easy to add multiple
experimental conditions. But under no circumstances should the
control group "do" anything (except "sit there").
Scott's design is a great example of exactly what a control group is
supposed to "do"--serve as a sanity check.
- Terry
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg