ietf-asrg
[Top] [All Lists]

[Asrg] 2. Analysis and characterization work

2003-09-02 06:41:35
On Fri, 29 Aug 2003 14:23:52 -0400, asrg-request(_at_)ietf(_dot_)org wrote:

Two things on my mind:

First off, I'd like to thank Scott Nelson for his recent posting 
regarding greylisting on spamtrap addresses.  His results underscore 
two important points that permate all of the group's 
analysis-and-characterization work:

1)  Although he wasn't trying to demonstrate a 
    reduction in spam *attempts*, his results
    serve as a pointed reminder of just how
    important a control condition is.  That is,
    if he *had* been trying to demonstrate a 
    reduction in spam attempts, the control
    condition ("do nothing," aka spam1) 
    actually "outperformed" the experimental 
    condition (aka spam2) by a measurable 
    (though not statistically significant) 
    margin.

2)  His data, which used a 3-week basis for
    for measurement, were volatile enough to
    preclude statistically significant 
    findings.  (Again, not that he was 
    looking for them, but object lessons 
    never hurt.)
    
The second thing on my mind regards the role of a control condition 
in experimental design.  I wonder if I've done an effective job of 
articulating just how important it is to make sure that the *only* 
systematic difference between (an) experimental group(s) and a 
control group is/are the independent variable(s).  I'm thinking here 
of the posting that suggested that control group addresses should 
open spam, click-thru, etc.

*Exactly the opposite is true*.  A control group is just like a 
"sugar pill" condition in a drug study.  The "job" of a control group 
is to show the effects of doing *nothing*.  Now, if one is interested 
in testing the effects of click-thru, it's easy to add multiple 
experimental conditions.  But under no circumstances should the 
control group "do" anything (except "sit there").

Scott's design is a great example of exactly what a control group is 
supposed to "do"--serve as a sanity check.

- Terry



_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg