Re: [Asrg] Re: 3b. SMTP Verification - Reputation Systems and their Prob

Anne P. Mitchell, Esq. wrote:
> Oops...1000 apologies - resent with proper subject line:
NP :)

> > > why would any proposed
> > > reputation or accrediation systems of the future be any different?
>
> We're getting a great deal of positive response to our IADB (ISIPP 
> Accreditation Database) even though we haven't publicly announced it yet 
> (if you are not familiar with it you can read about it at 
> http://www.isipp.com/iadb.php).  What are the problems you feel are 
> innate to DNS blocklists which you feel are also ported to DNSWLs or 
> other non-block DNS lists?
Allow me to explain. There are multiple problems with reputation and 
accreditation systems, virtually all of which are non-technical in 
nature. There are a few technical problems as well: suspectibility to 
DDOS attacks unless the database is distributed and not centralized, 
whether automatic testing for things like open relays is foolproof, 
problems with protocols, etc. However I want to concentrate on the 
non-technical issues.
There are multiple types of reputation and accreditation systems (please 
feel free to correct anything). At the most basic, a reputation system 
provides information about a given organization, MTA, IP, ISP, etc. such 
as how long has this system been an MTA, average amount of outgoing 
email per day, etc. This information is not biased towards or away from 
blacklisting, but rather is provided as simply a source of statistical 
data to be used. This is similar to the statistical data provided by a 
credit agency on a credit report in the US about the accounts a person 
has, amount of credit, average balance, etc. The closest example of this 
today is SenderBase.
Then we have the various blacklist reputation systems which provide 
negative information such as this IP is an open relay, this IP is a 
spammer, this ISP hosts spammers, etc. This is the most prevalent type 
today, examples abound (SPEWS, MAPS, etc.). A real world example of this 
would be a credir agency reporting negative data about a person such as 
bankruptcy, bad credit, collections, etc. Human or community-based 
systems such as SpamCop or CloudMark would fall into this category as well.
Now we also have whitelist systems which are not reputation systems per 
se, but rather are accreditation systems. They allow parties with 
otherwise bad or unknown reputation such as new providers, IP blocks 
reassigned from spammers, email that would otherwise be caught by a 
filter, etc. to be accredited as good guys. Examples of this are 
BondedSender and your IADB. An example of this in the real world would 
be a person with bad or unknown credit coming to get a loan, and 
bringing along a letter of reference or a letter vouching for him from a 
trusted third party.
I am not saying that they are used specifically by bad guys or people 
with bad prior reputation, but the main purpose of accreditation systems 
is whitelisting email that might not get through otherwise.
There are also differences how different types of systems are managed - 
some are based on automated testing of open relays and such (SpamHaus's 
XBL, MAPS RSS and OPS, etc.), others are based on spam reports from 
people or community (SpamCop, Cloudmark, etc.), yet many are also based 
specifically on human reports and decisions by operators (SpamHaus 
ROKSO, SPEWS, etc.). The SMTP VERIFY subgroup has also discussed other 
posibilities such as reputation based on "web of trust".
The major problem with all of these is management rather than the 
technical details. Let me go through these:
1. Statistical systems - for these there are several problems. First 
problem is accuracy - the data collected by the system must be accurate 
and in order to assure that the collection process needs to be done in a 
 statistically sound matter. In particular, such system needs to cover 
a statistically significant percentage of email traffic to be accurate. 
Second, problem with statistical systems is lack of openness - the 
collection process and internal management practices must be open for 
public scrutiny and preferablly audited on a regular basis by an 
independent third party. This will preclude the operators from skewing 
the data based on their own motives. Third problem, is the "tyranny by 
the mob" - a statistical system must be able to account for the 
possibility of multiple reporting systems ganging up together and 
reporting false data in order to destroy someone's reputation. The 
fourth problem is legal, which I am not sure if it would apply. Anne, as 
a lawyer is more qualified to answer whether such statistical systems 
are likely to get sued. The fifth problem is whether such systems raise 
the bar for new MTAs on the Internet.
2. Blacklist reputation services - there are multiple problems which 
vary depending on how the list is managed. The main problem common to 
all of these is lack of open procedures about how things get added to 
the list, how long they stay on and how they can be taken off. Another 
big problem is lack of review or any kind of appeal process unlike the 
credit agencies in the US which was my real world example. There are 
also subtle differences between management types - human and community 
based systems are suspectible to the "tyranny by the mob" problem 
described above, automated systems need to deal with dynamic IPs and 
clarify the length of time spend on list and re-testing intervals (AOL 
retests once a day, not all BLs do), some BL operators tend to list 
others on the same block or ISP causing collateral damage, etc. Legal 
issues very much apply here as we saw from the various lawsuits 
surrounding blacklists. Also, in many cases blacklist operators are 
making a decision based on their beliefs which might not the same as the 
beliefs of their users. Many ISPs would rather make the decision 
themselves if they had access to the raw data that the BL operators had. 
That raw data in many cases is not exposed, or cannot be provided in an 
automated form. Lack of any appeal process, especially with SPEWS is a 
big problem as well. Lack of openness in BL operations is an issue. No 
third party audit or review to let us know that the BL operators are 
following their guidelines.
3. Whitelist and Accreditation services - less problems than others and 
more subtle. The main issue is one of a gatekeeper - if a small set of 
whitelists or one is choosen by many large ISPs, that whitelist 
effectively becomes the gatekeeper for the email system - "power 
corrupts, absolute power corrups absolutely". In order to counteract 
that, whitelists need to be very open and auditable by third parties. Of 
course, currently since we do not have too many of those, it's pretty 
earlier to say that, but the issue is important to keep in mind. DDOS 
issues come to mind as well.
In practice it is impossible for an average spam filter to utilize data 
from too many reputation systems, and therefore it is expected that only 
a few will be choosen. That would mean we must be careful with whitelist 
services to make sure that they do not raise the barrier of entry to the 
Internet community.
Yakov




_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg