Chris Lewis writes:
Justin Mason wrote:
Chris said:
Okay, suggest how a spam/ham collection can be used to measure the
effectiveness of the following techniques that are, or can be, used in
an anti-spam solution:
1) grey listing
2) sender/sender domain verification
3) Challenge/response
4) SPF and DKIM
5) PKI
6) CSV
7) Non-existant users
8) DCC or other distributed checksumming methodologies.
Similarly DNSBLs.
The effectiveness of all of these, except possibly for greylisting and
C/R, can be measured accurately. We are doing it in SpamAssassin ;) --
here's how.
First, you accept every message, and record the "real-time" data
points regarding how the message was listed against those services,
and/or how it *would* have been rejected at SMTP transaction time (if
at all). However, you don't reject, you accept everything.
Then, later, provide a way for hand-sorting to take place, and compare
the results of the hand-sorting with what the various other techniques
would have done with those messages.
Hey presto, you've now got a way to compare accuracy and effectiveness
of those techniques! Simple as that.
Do you see any provision for that with _any_ of the spam/ham
collections? Can you do that with a pre-existing spam/ham collection if
the technique you were trying to test _wasn't_ being collected at the
time the spam/ham collection was being made?
This points up my very point: in order to do a proper sampling of the
effectiveness of a technique is that you do it in real time, at the time
the emails were sent. In other words, on real mail streams.
Ah, if you're talking specifically about *pre-existing* spam/ham
collections, then it's correct to say that they won't help, no. To
measure network test efficacy, you need to create the spam/ham collection
in real-time, during the test.
--j.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg