Chris Lewis said:
Got a number of signs of interest in my metrics, so, I thought I'd spend
a little more time on them. Added a few of the more famous BLs we don't
use, and "fixed" our whitelist metrics so they're useful in determining
relative effectiveness in getting entries removed (which includes _our_
effectiveness in being able to request a retest).
While we're at it, then, I'll post the SpamAssassin effectiveness
measurements from the most recent GA run (Feb 16 2003).
These are taken from a mail corpus of 63203 spam messages and 83465
non-spam messages, taken as a representative, hand-sorted sample from
about 15-20 people's mail feed over the 6 months before Feb 16 (spam from
only 3 months). So it's not live, but it's reasonably close.
Each line consists of several figures, like so:
total% spam% nonspam% soratio rank score name
2.752 6.3763 0.0072 0.999 0.94 4.29 RCVD_IN_OPM
spam% and nonspam% are the most important ones in this case; it indicates
the percentages of the spam or nonspam corpora that were hit by the rule.
"soratio" is the "spam over overall" ratio -- so the probability that a
mail hit by this rule is spam. A rule with an soratio of 1.0 is a perfect
rule for finding spam, and 0.0 is a perfect rule for nonspam.
They're sorted by the rule's "rank", which is effectively accuracy (as
per s/o ratio) and number of hits combined. "score" is the score assigned
inside SpamAssassin 2.50.
First, the DNSBL effectiveness ratings:
2.752 6.3763 0.0072 0.999 0.94 4.29 RCVD_IN_OPM
3.468 8.0139 0.0264 0.997 0.94 1.10 RCVD_IN_SBL
0.751 1.7420 0.0012 0.999 0.93 4.29 RCVD_IN_DSBL
1.328 2.9097 0.1306 0.957 0.82 0.18 RCVD_IN_RELAYS_ORDB_ORG
18.118 33.6297 6.3715 0.841 0.60 1.21 RCVD_IN_NJABL
29.728 53.1889 11.9619 0.816 0.58 0.51 RCVD_IN_OSIRUSOFT_COM
5.883 8.7290 3.7273 0.701 0.33 1.25 RCVD_IN_RFCI
12.092 13.6607 10.9040 0.556 0.18 0.00 RCVD_IN_UNCONFIRMED_DSBL
and some others of general interest. Here's Razor 2's effectiveness
on our corpora. RAZOR2_CHECK is Razor's default check, while the CF_RANGE
tests are for Razor's "confidence factor" of 0 to 100.
19.503 45.1039 0.1162 0.997 0.99 1.11 RAZOR2_CF_RANGE_91_100
23.614 54.1794 0.4685 0.991 0.99 0.79 RAZOR2_CHECK
0.200 0.4367 0.0216 0.953 0.81 0.58 RAZOR2_CF_RANGE_31_40
0.147 0.3323 0.0072 0.979 0.88 0.85 RAZOR2_CF_RANGE_51_60
0.168 0.3797 0.0084 0.978 0.88 0.33 RAZOR2_CF_RANGE_81_90
0.822 1.8448 0.0467 0.975 0.87 0.98 RAZOR2_CF_RANGE_21_30
1.293 2.8986 0.0779 0.974 0.87 0.59 RAZOR2_CF_RANGE_11_20
0.278 0.6171 0.0216 0.966 0.84 0.37 RAZOR2_CF_RANGE_41_50
0.162 0.3528 0.0168 0.955 0.81 0.00 RAZOR2_CF_RANGE_61_70
0.595 1.2230 0.1198 0.911 0.71 0.27 RAZOR2_CF_RANGE_01_10
0.130 0.2468 0.0419 0.855 0.58 1.86 RAZOR2_CF_RANGE_71_80
next, Pyzor and DCC on our corpora:
4.535 10.4900 0.0264 0.997 0.94 1.25 PYZOR_CHECK
5.736 13.2367 0.0563 0.996 0.94 2.76 DCC_CHECK
Finally, Bayesian learning results, using SpamAssassin's auto-learning
alone:
27.135 0.0190 47.6679 0.000 1.00 -6.60 BAYES_01
20.542 47.5832 0.0659 0.999 1.00 2.85 BAYES_90
8.457 0.0016 14.8601 0.000 0.95 -6.40 BAYES_00
6.427 14.8996 0.0120 0.999 0.95 2.81 BAYES_80
7.458 0.0142 13.0953 0.001 0.95 -5.80 BAYES_10
3.479 8.0613 0.0084 0.999 0.94 2.79 BAYES_99
4.613 10.5660 0.1054 0.990 0.92 2.19 BAYES_70
3.376 0.0490 5.8959 0.008 0.92 -3.10 BAYES_20
3.417 0.2753 5.7952 0.045 0.82 -1.60 BAYES_30
3.560 7.4395 0.6218 0.923 0.74 1.16 BAYES_60
Notes:
SBL's effectiveness is 8.01% for us; for Chris it's 7.63%. Pretty
close. But our effectiveness for OPM is much lower, at 6.3% instead
of 51%.
Chris notes an accuracy rate of 20% for SPEWS. We get 53% (as
relays.osirusoft.com), but with 11% FPs.
--j.
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg