RE: [spf-discuss] Making spam scores public

At 11:42 PM 3/6/2007 -0600, Seth Goodman wrote:
> David MacQuigg wrote on Monday, March 05, 2007 4:27 PM -0600:
>
> > It seems to me the real goal of our work on these reputation systems
> > is to provide a universal solution to the spam problem.
>
> While laudable, this is not possible.
Nattering Nabob of Negativism!!! :>)
http://en.wikipedia.org/wiki/Spiro_Agnew

> > Keeping the data private means there is no fundamental difference
> > between what you are doing and what any large ISP or spam-appliance
> > company does.  How can you expect your solution to be any better
> > than what these private companies are doing?
>
> Having private data is a large advantage, which is why large ISP's don't
> publish their internal listing criteria.  Attacking a small network that
> uses local reputation data followed by Bayesian content filtering is
> inherently harder than attacking a system using public DNSBL's and
> content filters with public rule sets.  You can get some unwanted
> messages through, but you can't test the messages for deliverability
> ahead of time.  The only commonality among networks that use local data
> is the code that generates it.  The data itself, and the system
> parameters that drive the decisions, is all unknown to attackers.
I think we need to make a distinction between private reputation data and 
private rule sets or methods.  I can see your point if we are talking about 
rule sets and methods, and I am inclined to agree, but not entirely 
convinced.  The SpamAssassin folks argue that making their rule sets public 
is not a problem.  If I understand their argument, it is that their rule 
sets are so large and hard to work around that it takes spammers months to 
adapt, and by that time, the next update on their rule set is available.  I 
wish I had a link for you, but I do recall seeing a graph of spam vs time 
supporting this argument.  Whatever the conclusion, it doesn't really 
matter, because we use SpamAssassin only to process messages that are not 
whitelisted, and as a means of generating reputation scores for domains 
that have not yet qualified to be whitelisted.  SpamAssassin scores are 
plenty accurate for that purpose.
If spammers get very good at avoiding SpamAssassin's rule sets, we can 
switch to another filter, or even use different filters for different 
receivers, and keep the choices secret.  As long as we get some feedback on 
the filter's decisions, we can include this feedback in an overall average 
rating for a domain.
As to the tactical advantage of keeping the reputation data secret, I see 
none.  These are long-term averages of results from many receivers, not a 
rapid-feedback loop to help spammers improve their methods.  On the 
contrary, I see a large advantage to publishing the data.  This is what 
will motivate legitimate senders to block the zombies in their networks by 
publishing better authentication records.
When email recipients can see a direct comparison of spam ratings for 
comcast.net and aol.com, Comcast might just decide that publishing a strict 
authentication record would be in their best interest.  They might lose 
some of their spamming customers, but so will every other large ISP that 
tolerates spammers.  My guess is that they would welcome an opportunity to 
tell their spammers - "Hey guys, we have to do this.  It's not our fault."
> The private data advantage is reduced if your incoming mail flow is too
> small.  Communications among a few peer systems can help greatly in this
> case.
The main problem I see with private data is that it doesn't allow for the 
kind of rapid global communication needed to make spamming unprofitable.  I 
realize that peers can be located anywhere in the world, so when I say 
"global", I don't mean geography.  I mean no isolated "islands" of peers 
that can be attacked one at a time.  If it takes even a few hours to 
downgrade a reputation, that will be plenty of time for spammers to 
inundate one island, then move on to the next.
If I understand the Gossip system, reputation information "diffuses" 
throughout the network of peers, one link at a time.  How long will it take 
before the whole world knows that a particular domain has been taken over 
by spammers?
> As an aside, Bayesian filters don't necessarily work better than
> carefully maintained rule sets, but they do it with a fraction of the
> maintenance.  Private reputation data created from your own mail flow
> holds the same promise.
Bayesian filters, heuristic rulesets, IP blacklists, all are inferior to 
feedback from recipients.  The trick is to make the amount of spam from 
whitelisted senders small enough that recipients don't mind having to 
report it.  In the last three weeks, I've seen only 5 whitelisted spams in 
my inbox, 3 from google.com, and 2 from comcast.net.  With only one or two 
spams a week, recipients won't mind dropping what they are doing, quickly 
reviewing the content of the message, and forwarding it to a spam-reporting 
address.
We can also make things nicer for recipients by sending them an immediate 
acknowledgement of their report, and a link to a website where they can see 
their report listed along with any others for the domain in question, the 
response of the domain postmaster, and any actions taken by the Rating 
Services watching this domain.
A few months ago, I saw a burst of spam from Yahoo's webmail servers 
lasting a few days.  I expect they will be much quicker in shutting down 
these sources when they are prodded by our spam reports, and when all they 
have to do is publish one DNS record.
> I don't mean to imply that there is no use for public reputation data.
> Evaluating whether to use data from a particular source means knowing
> who they are.  This exposes them to legal action, a risk most companies
> do not want.  An alternative is creating composite data from all
> submitters, which is the SpamCop approach that many sites find too
> unreliable.  In the end, the most successful public lists are created
> from networks of trusted private sources and are carefully managed.
Our ratings will come from many sources, including Gossip, if we can find a 
way to interface.  The simplest system, which we are testing now on a small 
scale, simply takes an "average" of the SpamAssassin scores from many 
receivers over a long period of time, discarding the "outliers", which we 
define to be any source that attempts to move the average too much in 
either direction.  This will eliminate the most obvious attack, sending 
huge volumes of phony mail to a collaborating recipient, so as to drive up 
the "ham" count.
While I hope that spammers will simply give up, and not force us to the 
next step, I am fully prepared for a battle of wits, as clever spammers try 
to fool equally clever managers at the Rating Services using our 
Registry.  I'm working now on some Python scripts to display the data on a 
domain in a way that will allow managers to quickly spot anomalies.  It 
should be very difficult for a spammer to generate a broad distribution of 
"ham" over a long period of time, enough to look like a normal legitimate 
sender.
I still don't understand the legal threat you keep referring to.  There is 
no such thing as a "bad" reputation in our system.  Ratings range from C 
(unknown) to A (less than one spam in 100 messages).  We don't bother with 
lower ratings, because we assume that no spammer will continue to use a 
name with a rating lower than a fresh new "unknown" name.  If a spammer is 
thwarted in an attempt to gain a higher reputation, who is he going to 
sue?  What would be the allegation - "Spamhaus, you failed to give me the 
A-rating I deserve after 3 months of diligently faking a legitimate mail-flow?"
I don't see any threat from legitimate senders who lose their reputation 
through innocent mistakes.  A well-managed rating service will work with a 
legitimate mailer to correct the mistake quickly.  Let's say yahoo.com is 
doing quite well with their current default record, authorizing 84992 IP 
addresses.  Suddenly spammers discover that they can forge Yahoo's name, at 
least on the zombies that lie within one of these huge IP blocks.  What 
will Yahoo do, hire lawyers to sue rating services all over the world, or 
simply assert control of their Registry record, and de-authorize the zombies?
> > I think the way to deal with threats of costly lawsuits is to set up
> > the company in a jurisdiction with more common sense in their legal
> > system than the USA.
>
> This is the precise reason that U.S. companies will not likely make
> their reputation data public.
The exception would be large companies like Ironport.  No spammer would 
dare sue them.  They do in fact, make their data public, just not in a way 
that can be automated without paying a fee.  I expect that fees to rating 
services like Ironport will be the biggest cost in providing Registry 
services.  That is as it should be.  We need the best services in the world 
to provide the most reliable domain ratings.  Everything else can be automated.
I believe the reason we don't have public reputation data is not fear of 
lawsuits, but rather a desire by companies to maintain a competitive 
advantage in selling their bundled products.
> Even if there were no threat of lawsuits,
> publishing this data tells your attackers how effective they were with
> each spam run.
The data that is published is long-term averages of data from many 
sources.  This will be very little value to the spammer.  The only 
immediate feedback a spammer might see is an alert that goes out when a 
reputable domain is suddenly hijacked.
> > If some rating service is put out of business by a lawsuit, others
> > will take its place.
>
> Even the threat of lawsuits is enough to deter most people.
The few Rating Services that are brave enough to not fear harassment in 
U.S. courts, will include the ones listed in our Registry records.
A much bigger worry regarding the reliability of Rating Services will be 
the possibility of bogus services controlled by spammers.  Our strategy 
here is to pick the best services by allowing Registry subscribers to 
designate what fraction of their subscription fee goes to each 
Service.  Corrupt or incompetent services will quickly lose their income, 
and eventually be dropped from the Registry.
-- Dave


-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription, 
please go to http://v2.listbox.com/member/?list_id=735