At 5:42 PM -0700 2003/08/12, Justin Mason wrote:
We (SpamAssassin) measure FP/FN rates for DNSBLs as part of our rescoring
process. However the results aren't prettified for the web. Maybe
sometime soon.
Scroll down on this page:
http://useast.spamassassin.org/dist/rules/STATISTICS-set1.txt
Interesting. Taking this data and sorting it by the accuracy, we get:
OVERALL% SPAM% HAM% S/O RANK SCORE NAME
15.868 22.8473 0.0040 1.000 0.95 4.30 RCVD_IN_OPM
12.937 18.6272 0.0026 1.000 0.94 4.30 RCVD_IN_OPM_HTTP
10.065 14.4926 0.0007 1.000 0.94 4.30 RCVD_IN_OPM_HTTP_POST
8.070 11.6201 0.0013 1.000 0.93 4.30 RCVD_IN_OPM_SOCKS
1.104 1.5904 0.0000 1.000 0.92 4.30 RCVD_IN_OPM_WINGATE
0.029 0.0424 0.0000 1.000 0.92 2.80 RCVD_IN_SORBS_WEB
0.015 0.0212 0.0000 1.000 0.92 2.80 RCVD_IN_OPM_ROUTER
0.001 0.0015 0.0000 1.000 0.92 0.00 RCVD_IN_NJABL_MULTI
19.465 28.0124 0.0363 0.999 0.95 1.10 RCVD_IN_OSIRU_PROXY
3.903 5.6145 0.0132 0.998 0.92 1.10 RCVD_IN_SORBS_SMTP
31.969 45.9802 0.1209 0.997 0.97 1.10 RCVD_IN_SORBS_HTTP
23.752 34.1549 0.1057 0.997 0.96 1.10 RCVD_IN_SORBS_MISC
39.058 56.1503 0.2042 0.996 0.98 1.10 RCVD_IN_DSBL
27.132 39.0053 0.1441 0.996 0.96 1.10 RCVD_IN_NJABL_PROXY
1.783 2.5591 0.0192 0.993 0.90 0.53 RCVD_IN_NJABL_DIALUP
1.022 1.4665 0.0112 0.992 0.90 0.85 RCVD_IN_OSIRU_DIALUP
12.520 17.9202 0.2439 0.987 0.91 1.27 RCVD_IN_SBL
9.706 13.8724 0.2353 0.983 0.89 0.93 RCVD_IN_OSIRU_SPAMWARE
9.036 12.9098 0.2320 0.982 0.89 0.64 RCVD_IN_NJABL_SPAM
3.435 4.8891 0.1302 0.974 0.85 0.00 RCVD_IN_OSIRU_RELAY
3.143 4.4719 0.1209 0.974 0.85 2.55 RCVD_IN_DYNABLOCK
0.361 0.5108 0.0192 0.964 0.82 2.60 RCVD_IN_SORBS_ZOMBIE
41.526 58.5996 2.7169 0.956 0.88 0.10 RCVD_IN_SORBS
41.161 57.6715 3.6322 0.941 0.84 0.10 RCVD_IN_NJABL
11.074 15.4265 1.1810 0.929 0.76 0.10 RCVD_IN_RFCI
3.185 4.4132 0.3945 0.918 0.72 1.31 RCVD_IN_NJABL_RELAY
3.802 5.2075 0.6060 0.896 0.67 0.00 RCVD_IN_SORBS_SPAM
38.389 51.8980 7.6802 0.871 0.68 0.00 RCVD_IN_OSIRU
0.056 0.0718 0.0185 0.795 0.46 1.52 RCVD_IN_SORBS_SOCKS
11.116 13.7768 5.0683 0.731 0.38 0.00 RCVD_IN_OSIRU_SPAM_SRC
0.016 0.0180 0.0106 0.630 0.23 0.00 RCVD_IN_NJABL_CGI
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_SORBS_BLOCK
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RSS
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RBL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_NML
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_DUL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_BL_SPAMCOP_NET
0.082 0.0026 0.2630 0.010 0.89 -0.10 RCVD_IN_BSP_OTHER
0.359 0.0087 1.1546 0.007 0.90 -4.30 RCVD_IN_BSP_TRUSTED
And sorting by the "ham" percentage:
OVERALL% SPAM% HAM% S/O RANK SCORE NAME
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_BL_SPAMCOP_NET
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_DUL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_NML
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RBL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RSS
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_SORBS_BLOCK
0.001 0.0015 0.0000 1.000 0.92 0.00 RCVD_IN_NJABL_MULTI
0.015 0.0212 0.0000 1.000 0.92 2.80 RCVD_IN_OPM_ROUTER
0.029 0.0424 0.0000 1.000 0.92 2.80 RCVD_IN_SORBS_WEB
1.104 1.5904 0.0000 1.000 0.92 4.30 RCVD_IN_OPM_WINGATE
10.065 14.4926 0.0007 1.000 0.94 4.30 RCVD_IN_OPM_HTTP_POST
8.070 11.6201 0.0013 1.000 0.93 4.30 RCVD_IN_OPM_SOCKS
12.937 18.6272 0.0026 1.000 0.94 4.30 RCVD_IN_OPM_HTTP
15.868 22.8473 0.0040 1.000 0.95 4.30 RCVD_IN_OPM
0.016 0.0180 0.0106 0.630 0.23 0.00 RCVD_IN_NJABL_CGI
1.022 1.4665 0.0112 0.992 0.90 0.85 RCVD_IN_OSIRU_DIALUP
3.903 5.6145 0.0132 0.998 0.92 1.10 RCVD_IN_SORBS_SMTP
0.056 0.0718 0.0185 0.795 0.46 1.52 RCVD_IN_SORBS_SOCKS
0.361 0.5108 0.0192 0.964 0.82 2.60 RCVD_IN_SORBS_ZOMBIE
1.783 2.5591 0.0192 0.993 0.90 0.53 RCVD_IN_NJABL_DIALUP
19.465 28.0124 0.0363 0.999 0.95 1.10 RCVD_IN_OSIRU_PROXY
23.752 34.1549 0.1057 0.997 0.96 1.10 RCVD_IN_SORBS_MISC
3.143 4.4719 0.1209 0.974 0.85 2.55 RCVD_IN_DYNABLOCK
31.969 45.9802 0.1209 0.997 0.97 1.10 RCVD_IN_SORBS_HTTP
3.435 4.8891 0.1302 0.974 0.85 0.00 RCVD_IN_OSIRU_RELAY
27.132 39.0053 0.1441 0.996 0.96 1.10 RCVD_IN_NJABL_PROXY
39.058 56.1503 0.2042 0.996 0.98 1.10 RCVD_IN_DSBL
9.036 12.9098 0.2320 0.982 0.89 0.64 RCVD_IN_NJABL_SPAM
9.706 13.8724 0.2353 0.983 0.89 0.93 RCVD_IN_OSIRU_SPAMWARE
12.520 17.9202 0.2439 0.987 0.91 1.27 RCVD_IN_SBL
0.082 0.0026 0.2630 0.010 0.89 -0.10 RCVD_IN_BSP_OTHER
3.185 4.4132 0.3945 0.918 0.72 1.31 RCVD_IN_NJABL_RELAY
3.802 5.2075 0.6060 0.896 0.67 0.00 RCVD_IN_SORBS_SPAM
0.359 0.0087 1.1546 0.007 0.90 -4.30 RCVD_IN_BSP_TRUSTED
11.074 15.4265 1.1810 0.929 0.76 0.10 RCVD_IN_RFCI
41.526 58.5996 2.7169 0.956 0.88 0.10 RCVD_IN_SORBS
41.161 57.6715 3.6322 0.941 0.84 0.10 RCVD_IN_NJABL
11.116 13.7768 5.0683 0.731 0.38 0.00 RCVD_IN_OSIRU_SPAM_SRC
38.389 51.8980 7.6802 0.871 0.68 0.00 RCVD_IN_OSIRU
And then by score:
OVERALL% SPAM% HAM% S/O RANK SCORE NAME
15.868 22.8473 0.0040 1.000 0.95 4.30 RCVD_IN_OPM
12.937 18.6272 0.0026 1.000 0.94 4.30 RCVD_IN_OPM_HTTP
10.065 14.4926 0.0007 1.000 0.94 4.30 RCVD_IN_OPM_HTTP_POST
8.070 11.6201 0.0013 1.000 0.93 4.30 RCVD_IN_OPM_SOCKS
1.104 1.5904 0.0000 1.000 0.92 4.30 RCVD_IN_OPM_WINGATE
0.029 0.0424 0.0000 1.000 0.92 2.80 RCVD_IN_SORBS_WEB
0.015 0.0212 0.0000 1.000 0.92 2.80 RCVD_IN_OPM_ROUTER
0.361 0.5108 0.0192 0.964 0.82 2.60 RCVD_IN_SORBS_ZOMBIE
3.143 4.4719 0.1209 0.974 0.85 2.55 RCVD_IN_DYNABLOCK
0.056 0.0718 0.0185 0.795 0.46 1.52 RCVD_IN_SORBS_SOCKS
3.185 4.4132 0.3945 0.918 0.72 1.31 RCVD_IN_NJABL_RELAY
12.520 17.9202 0.2439 0.987 0.91 1.27 RCVD_IN_SBL
39.058 56.1503 0.2042 0.996 0.98 1.10 RCVD_IN_DSBL
31.969 45.9802 0.1209 0.997 0.97 1.10 RCVD_IN_SORBS_HTTP
27.132 39.0053 0.1441 0.996 0.96 1.10 RCVD_IN_NJABL_PROXY
23.752 34.1549 0.1057 0.997 0.96 1.10 RCVD_IN_SORBS_MISC
19.465 28.0124 0.0363 0.999 0.95 1.10 RCVD_IN_OSIRU_PROXY
3.903 5.6145 0.0132 0.998 0.92 1.10 RCVD_IN_SORBS_SMTP
9.706 13.8724 0.2353 0.983 0.89 0.93 RCVD_IN_OSIRU_SPAMWARE
1.022 1.4665 0.0112 0.992 0.90 0.85 RCVD_IN_OSIRU_DIALUP
9.036 12.9098 0.2320 0.982 0.89 0.64 RCVD_IN_NJABL_SPAM
1.783 2.5591 0.0192 0.993 0.90 0.53 RCVD_IN_NJABL_DIALUP
41.526 58.5996 2.7169 0.956 0.88 0.10 RCVD_IN_SORBS
41.161 57.6715 3.6322 0.941 0.84 0.10 RCVD_IN_NJABL
11.074 15.4265 1.1810 0.929 0.76 0.10 RCVD_IN_RFCI
38.389 51.8980 7.6802 0.871 0.68 0.00 RCVD_IN_OSIRU
11.116 13.7768 5.0683 0.731 0.38 0.00 RCVD_IN_OSIRU_SPAM_SRC
3.802 5.2075 0.6060 0.896 0.67 0.00 RCVD_IN_SORBS_SPAM
3.435 4.8891 0.1302 0.974 0.85 0.00 RCVD_IN_OSIRU_RELAY
0.016 0.0180 0.0106 0.630 0.23 0.00 RCVD_IN_NJABL_CGI
0.001 0.0015 0.0000 1.000 0.92 0.00 RCVD_IN_NJABL_MULTI
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_SORBS_BLOCK
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RSS
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_RBL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_NML
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_MAPS_DUL
0.000 0.0000 0.0000 0.500 0.11 0.00 RCVD_IN_BL_SPAMCOP_NET
0.082 0.0026 0.2630 0.010 0.89 -0.10 RCVD_IN_BSP_OTHER
0.359 0.0087 1.1546 0.007 0.90 -4.30 RCVD_IN_BSP_TRUSTED
I note that several of the black lists have zero hits on both the
"spam" and "ham" categories. Is this because you do not have access
to them on this machine, and therefore cannot measure their
effectiveness and accuracy? Of course, the lists with a negative
score should be white listed.
What about measures of unique hits? I.e., a particular IP is
found only in a single black list? Is there some other way to
measure hit coverage? It would be interesting to know what the
minimal set of black lists would be to get the maximum coverage....
Thanks!
--
Brad Knowles, <brad(_dot_)knowles(_at_)skynet(_dot_)be>
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg