Summary of the ASRG discussion on draft-church-dnsbl-harmful-01.txt as of 04-05-2006. This is a review of the draft-church-dnsbl-harmful-01.txt document as discussed on the ASRG mailing list http://news.gmane.org/gmane.ietf.asrg/ Note: Some of the posts are missing from the archive, such as Dan Oetting's message-id <84E9C57C-1CE6-4232-90FC-7BE5F9EEC422(_at_)qwest(_dot_)net>. I was going to refer to archive urls but ended up just giving Message-IDs. The church document has the following structure. 1) An informal description of IP DNS blacklists. 2) A criticism of the policies and goals of DNS blacklist operators. 3) A claim of damage to third party mail transport. 4) A claim that botnets cause DNS blacklists to be out of date immediately. 5) An experiment on the accuracy of filtering using DNS blacklists. 6) A suggestion for the improvement of IP recognition 7) A moral argument of responsibility by DNS blacklist operators. 8) A suggestion to deprecate parts of the internet mail system to newer technologies. Taken together, the author comes to the conclusion that DNS blacklists should not be used. The ASRG discussion centered on several of these points. Claims 2) and 3) are sensitive issues fraught with political aspects, and are therefore not appropriate for technical criticism, although they do represent some valid concerns that are often brought up. Dan Oetting's thread deals with 4). There is agreement that botnets are a problem, but there is some research into the problem. Jim MacLeod thread makes these points: Regarding 4) IP assignments to particular computers tend to change slowly enough (on the order of a month) to make IP lists valuable. [reference?} Regarding 2), diversity is a strength in anti-spam. Regarding 5), the experiment is flawed due to its size. Daniel Feenberg mentions the increased legitimacy of rejecting mail at the SMTP transaction protocol level rather than silently discarding it for filtering reasons. Unlike content filtering, DNSBLs can be used at the transaction time. Steve Atkins <4D2D1BA3-767A-47FC-BC9F-D54829C2E450(_at_)blighty(_dot_)com> suggests there are significant general problems with DNSBLs, such as being misused as reputation systems, technical flaws in their usage, etc. It may be (Chris Lewis <442C2405(_dot_)1040601(_at_)americasm01(_dot_)nt(_dot_)com>) that the real problem is that people responsible for choosing DNSBLs aren't necessarily exercising due diligence and knowing what they're getting. Tony Finch disagrees with Steve Atkins, calling DNSBLs crucial. Nick Nicholas asked for solid evidence, thereby being relevant to 5). There is a big thread on testing philosophy of DNSBLs. Everyone agrees that 5) in Church's paper is flawed. The flaws are as follows: Flaw: The Church sample size (1,374 msgs) is much too small to be meaningful. Chris Lewis <442EE80C(_dot_)5080500(_at_)americasm01(_dot_)nt(_dot_)com> disagrees with Church's conclusions based on 65,000 users and 600,000 emails per day at Nortel. Flaw: the comparison of filters in Church's table is inapplicable. Steve Atkins <2BC9207C-3E51-4B88-97A8-7445E9A1313B(_at_)blighty(_dot_)com> points out that pretending that a DNSBL filter was used for comparison purposes, as is done by Church, is meaningless when it actually wasn't, because filtering during the SMTP transaction gives feedback (Daniel Feenberg's point) and therefore changes the spammer's behaviour. Flaw: Church doesn't explicitly define his terms, such as just what he considers spam. In turn, this means derived definitions of FP, FN, TP etc. are not fully mapped out. Laird Breyer and Chris Lewis discuss the consequences of using different definitions, resulting in contradictory claims. Flaw: Church's experiment is applicable at best to one type of email user. Chris Lewis's experience strongly contradicts the Church conclusion, and is based on very different business user email patterns. Flaw: Church's table data suggests rejecting DNSBLs individually on accuracy grounds. However, this is flawed in that DNSBLs are often used in conjunction with other filters. The table doesn't address these other possibilities at all. Several people point out that DNSBLs are meant to be used as part of a mix of techniques, ie that the whole is bigger than the sum of its parts. Peter J. Holzer <20060331191807(_dot_)GA14686(_at_)teal(_dot_)hjp(_dot_)at> offers some statistics on the most effective elements of his mix, others chime in on that thread. Walter Dnes <20060401015927(_dot_)GA27452(_at_)waltdnes(_dot_)org> addresses 2) and 3), which are out of scope of the technical criticism. He points out that 5) is flawed because it compares a DNSBL-only solution against an unnamed, unknown custom method, and lists several other flaws above. This is not realistic since DNSBLs are used in conjuction with other methods. Walter also points out that 8) is ridiculous. Thomas Choi <44322933(_dot_)7060103(_at_)nortel(_dot_)com> summarizes various issues that appeared during the discussion. Regarding 2) and 3), it's important for users of DNSBLs to apply diligence and caution. A single DNSBL verdict should never be used on its own, as is implied in 5). Regarding 5), spam filtering statistics always depend on type of email mix being received. This varies depending upon the type of user (business or private individual) and the type of social interactions (frequent mail with regular correspondents or regular short exchanges with strangers), etc. Regarding 8), Thomas points out that mail technologies arise out of many different needs of users, and what is optimal for one person or entity in terms of usability, time, expertise, processing power etc. is not necessarily optimal for another.