Re: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

On Thu, 30 Mar 2006, Michael Thomas wrote:

> Daniel Feenberg wrote:
> > On Thu, 30 Mar 2006, Nick Nicholas wrote:
> >
> > > On Thursday, March 30, 2006 at 8:49 AM Tony Finch wrote:
> > >
> > > > DNSBLs are not only useful, they are crucial.
> > >
> > > I'm not disagreeing with you, in fact I agree wholeheartedly.  However,
> > > does any empirical data exist to support this assertion?  I think we
> >
> > What sort of data did you want to see? Something about the average
> > success rates and false positive rates over all DNSBLs? Or an analysis
> > of one or a couple? When I looked, I found that most DNSBLs blocked
> > little spam, but that the best were quite effective. Is it sufficient to
> > show that the best are well run, or do we have to show that all are?
> Does there not exist something like, oh say, BLreports that
> judges you on false positive/false negative, coverage, timeliness,
> etc?

I attempted something along these lines some time ago, it is posted at 
http://www.nber.org/sys-admin/dnsbl-comparison.html but some might 
consider it out of date. It is very difficult to count false positives, 
since they are rarely reported.
We are very satisfied with the DNSBL we chose (Spamhaus) and have found 
that at any one time out of the over a 1,000 correspondents we deal with 
regularly and would notice if we blocked, usually none are blocked, 
sometimes one is. Since blocked legitimate mail is always returned to the 
sender by the source MTA, that does not represent lost mail, the sender is 
made aware of the problem and given an alternative (a web page) with an 
explanation.
There is no question that there are worse blacklists, which hardly seems 
like a reason to eliminate the good ones. Overall Spamhaus blocks less 
mail than content filters I am aware of (Spamassassin and Google Mail), 
but we judge the benefit of no lost mail to compensate.
A better study of false positives would require a large corpus of known 
good mail for a diverse set of destinations, with connecting MTA IP 
addresses. One could query the DNSBLs for those IP addresses, and 
calculate the probability that a legitimate message would be blocked. But 
I haven't found a corpus of known good mail. One source would be email 
confirmations of mailing-list signups, if anyone would like to share that 
with me. The saved mail file of an individual isn't very 
representative even if it is large.
>                 Mike

_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg