[Asrg] Collaborative real time spam blocking

Greetings,

I do not know if this is the correct forum to address this issue.  
Please let me know if these issues are being addressed elsewhere.  What 
I am seeking is a means for coordinating a grass roots trust network for 
aggressive dynamic blocking of spammer IP addresses.    While there are 
many blacklists out there, they do not provide for mechanisms to 
facilitate real time blocking.  Standards are needed for collaborative 
blocking,  incident reporting and verification.  Spammers move to 
quickly for traditional blacklisting approaches be effective.
My experience blocking spam over the last month suggests that a 
collaborative dynamic approach could be quite effective.  It is 
remarkable what I am able to achieve all by myself.  It seems clear to 
me that  collaboratively we could actually end the spam problem.
With the exponential growth of spam, filtering it is not enough.  The 
spam processing on our email server overloaded the machine to the point 
where it was forced to reboot several times a day.  While our policy of 
discarding invalid addresses rather than rejecting them exasperated the 
problem in our case, it will not be very long, given the growth rate of 
spam, before all email servers on the Internet have to face a similar 
day of recconning.
For just over a month I have waged a personal war against spam.   On a 
single machine with less than 500 email accounts I have identified just 
about 200,000 spam emails from about 60,000 individual IP addresses 
defining about 3000 IP groups (of 256) having 3 or more spammer IP 
addresses.  I am rejecting a total of about 750,000 IP addresses.
All the 'experts' tell me black lists don't work any more because spammers are 
moving too fast and there are too many of them.  There is no simple solution 
but I believe an agreesive, combined, scalable, collaborative grass roots 
approach can work.


While I have succeeded in significantly reducing the spam load on the 
machine, I cannot alone stop spam in a timely manner or  impact the spam 
problem in general.  Every day I identify about 1500 new spammer IP 
addresses.  In many cases I block them immediately after receiving just 
one spam.  This prevents getting hundreds of spams from each address.
The trick is to identify a spammer IP as soon as possible and block it 
as soon as possible.  If this was done widely, the spammers would be out 
of business.    The system must agressively  identify and block 
spammers, but it also must be forgiving and heal as spammers move around 
the net.  If the incentives are there for Internet service providers to 
enforce anti-spam policies to avoid being blocked, and timely blocking 
can significantly reduce the profits from spamming, spam can be stopped.
Spam me once, shame on you!  Spam me twice, shame on me!

Trust networks are needed to avoid malicious blocking.  Participants 
should be able to negotiate the degree of  confirmation desired before 
blocking and choose trusted  and untrusted sources.  There need to be 
mechanisms for sharing black lists, whitelists, incidents, reprieve 
requests.  This should work on a grass roots level without any  central 
authority.  It can include standards for DNSBL services but will require 
additional web services.  It may also suggest extensions to the header 
standards.  It would also be useful to have a means of summarizing spam 
incidents by IP or group in terms of time frame, number of incidents, 
number of email addresses, and perhaps confidence.
In the remainder of this email  I provide additional details of how my 
system is working at present in anticipation of questions some of you 
may have.  You can see the system working in almost real time at:
http://Xanthus.Net/spam.cgi

I also introduce the use of "honey pot" email addresses for spam 
identification, which is an idea I have not seem elsewhere.  However, I 
don't see this as a standards issue at this point, but you may read on 
if you are interested.
Please let me know if there are any relevant activities in Collaborative 
real time spam blocking standards activities.
Thanks,

Jim
-------------------------------------------------------------
Notes on  my current spam blocking activities:
http://Xanthus.Net/spam.cgi

Rapid spam identification is essential for dynamic blocking to be 
effective.  The primary method is what I call "honey pot" email 
addresses mined from the mail logs.  These are non-existent users or 
accounts that have been out of use for many years.  I have collected 
thousands of these as our providers default policy was to not return 
them as undeliverable.  Most were invented by worms or spammers and 
forged from addresses that were subsequently harvested.
Since these users do not exist, they cannot solicit any email.  Any 
email to these addresses is considered unsolicited and the address of 
the SMTP host is blocked.   Spam filtering tools may be used to 
automatically confirm the spam nature of these emails before 
automatically blocking for accounts that might conceivably get a valid 
email.
A second  source is users IMAP folders named SpamBlock.  They save 
emails there to prevent future emails for them or other users.  
SpamBlock folders are processed every few minutes.   Those who use a 
SpamBlock folder must use it responsibly to insure they don't block 
anybody legit. A secondary spam filter could help avoid errors here as 
well.
I check a whitelist and then just add a REJECT to the access database 
for SENDMAIL.  Periodically I add REJECTs for IP groups of 256 with 
three or more spammer IP addresses.
In a month I've only had three user complaints, one that was due to my 
neglecting to put hotmail.com on my whitelist, the other  two due to 
users who filed non-spam emails in their SpamBlock folders.  There is no 
painless way to solve the spam problem. I believe that dealing with 
wrongly blocked addresses is the price we must pay if we really want to 
stop spam.  Affirmative action against spam means guilty until proven 
innocent, but we can be very forgiving, at least on fist offenses.
I realize this is a tough policy, but it is what is needed, I think, to 
stop the exponential growth of spam.  It is dynamic, so that with enough 
participants, spam could be stopped no matter how fast the spammers 
move.  Spamming would become unprofitable and ISP would be motivated to 
enforce anti-spam policies.
I also forgive individual IP addresses after 10 days or so, I may change 
that.  Many seem to never spam again, while others repeat 
periodically.   I do not forgive those with repeated incidents to 
multiple email addresses yet.   I will give them another chance to be 
good sooner or later.
I think a combination of approaches is needed to fight spam.  My 
approach is best suited for blocking the new spambots and spammer 
sites.   If widely employed it could have a major impact on the industry.
There is much more I want to do. 

I want to add auto real time reporting of spammers also.  The standards 
can be important here so I am happy to see an ASRG group in that area.   
I see some similarities in some of the desired functionality that might 
also be employed in supporting dynamic collaborative blocking.
It has been suggested that my lists can be compressed greatly using CIDR 
notation for firewall rules.  That could also allow much greater 
selectivity and effectiveness in IP group blocking using powers of two 
group sizes.   And CIDR would be the preferred format according to our 
Internet provider.   Let me know if you know of any tools that could 
help automate this.
I am not sure how much further I'll get in my own personal war against 
spam, I need to find a way to support my efforts...  In the mean time 
let me know if you see any potential collaboration.
Thanks,

Jim Whitescarver
jim(_at_)xanthus(_dot_)net
973-643-0920

_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg