Attached is a ~/.procmailrc fragment that evaluates e-mail header
construction of e-mail messages to determine the probability that an
e-mail message is spam.
The numbers were derived by analyzing two large e-mail repositories;
one containing only spam, the other containing only non-spam. Let A be
the number of messages in the spam archive that have a certain header
characteristic divided by the total number of messages in the spam
archive, and B be the number of messages in the non-spam archive that
have the same characteristic divided by the total number of messages
in the non-spam archive. (A is the likelihood of a spam message
containing the characteristic, and B is the likelihood of a false
positive.)
Do this for many characteristics, dividing A by B, and picking the
half dozen, or so, largest values to determine which characteristics
will be used.
Use the negative of the natural logarithm of B in the attached recipe.
Example:
For a spam archive, of 1000 messages, where 600 have a common
characteristic; A = 600 / 1000 = 0.6.
For a non-spam archive, of 2000 messages, where 2 have the same
common characteristic; B = 2 / 2000 = 0.001.
A / B = 0.6 / 0.001 = 600. If 600 is larger than any other A / B
values for the half dozen other characteristics, then one of the
other characteristics should be replaced with this one.
The value used in the recipe would be ln (B) = ln (0.001) =
-6.90775527898:
* 6.90775527898^0 test_for_some_characteristic
which should be included in the last recipe in the attached.
The way it works is that that values, (6.90775527898 in the example,)
for each characteristic that is true for a specific message are added
together. Since the values are logarithms, this sum is a product. And,
since the values represent probabilities of false positives, the
chances of a total false positive are reduced with each true
characteristic, significantly, (assuming the characteristics are
statistically independent.)
Note that no single characteristic (orbs, etc.,) is capable of
generating a false positive, resulting in an inappropriately trashed
message.
It turns out to be about 70-85 percent effective, (but one can tinker
with the sigma limits, and probably tweek a little more out of it.)
The characteristics used in the attached recipe require:
1) A sendmail.cf, (or equivalent,) configuration that puts
"... (unknown ... in a "Received: " record if RDNS fails
for the HELO.
2) Some method of verifying whether the IP address contained in a
"Received: " record is a black listed host. (I used,
http://www.johncon.com/john/receivedIP/index.html, but
rblcheck(1), etc., can be used just as well.)
3) The name of your smtp host, somesmtpserverdomain.com.
4) Your e-mail address, someone(_at_)somedomain(_dot_)com(_dot_)
but can be modified to suit.
John
--
John Conover conover(_at_)email(_dot_)rahul(_dot_)net
http://www.johncon.com/
######################################################################
#
# Save the trusted return address.
#
:0 whc
SENDER=| formail -rtzx To:
#
# Save the machine generated return address.
#
:0 whc
FROM=| formail -rzx To:
#
# If the IP address in any "Received: " record is black listed, set
# ORBS to 1, else 0.
#
# (Note: this uses the off line black list from:
# http://www.johncon.com/john/receivedIP/index.html. Use any convenient
# method to set ORBS 1 if black listed, 0 if not, like rblcheck(1),
# etc.)
#
ORBS="0"
#
:0
* ? test -f "${HOME}/.procmail.reject"
* ? /usr/local/bin/receivedIPdb "${HOME}/.procmail.reject"
{
ORBS="1"
}
#
# If the smtp server had to generate the "Message-Id: " record, and the
# message is not from someone in the smtp server's domain, set
# MSGIDGENERATED to 1, else 0.
#
MSGIDGENERATED="0"
#
:0
* -3^0
* 2^0 ^message-id:.*somesmtpserverdomain\.com
* 2^0 !^(from|reply-to):.*somesmtpserverdomain\.com
{
MSGIDGENERATED="1"
}
#
# Evaluate header construction.
#
:0
* 6.35899699817^0 !^to:
* 2.31011481332^0 !^(to|cc):(_dot_)*someone(_at_)somedomain\(_dot_)com
* 5.05971401614^0 ^received:.*\(unknown +.*by.*somesmtpserverdomain\.com
* 6.07131492861^0 ? test ${ORBS} != "0"
* 7.45760928973^0 ? test ${SENDER} != ${FROM}
* 3.87409035224^0 ? test ${MSGIDGENERATED} != "0"
* 16^0 ^x-advertisement:
* 16^0 ^subject:.*adv(ertise(ment)?.*)?(:|$)
{
HEADERSCORE = $=
#
# Sigma values:
#
# 1.84102164502 = ln (1 sigma false positive), 1 in 6.30297437513
# 3.78318433404 = ln (2 sigma false positive), 1 in 43.9557890318
# 6.60772622151 = ln (3 sigma false positive), 1 in 740.796695584
# 10.3601014878 = ln (4 sigma false positive), 1 in 31,574.3873622
# 15.0649983951 = ln (5 sigma false positive), 1 in 3,488,555.79111
#
# Greater than 5 sigma chance of false positive can be safely
# trashed. Raising the value from 5 sigma to 4 sigma increases
# spam rejection, at the risk of false positives.
#
:0
* -15.0649983951^0
* $$HEADERSCORE^0
/dev/null
#
# Greater than 1 sigma chance of false positive, but less than 5
# sigma, is filed in the junk folder for evaluation; less than 1
# sigma has a significant probability of being non-spam. Raising
# the value from 1 sigma to 0 increases spam rejection, at the
# risk of false positives, which will be filed in the junk folder.
# Decreasing the value to 2 sigma increases the risk of false
# negatives, which will be passed.
#
:0:
* -1.84102164502^0
* $$HEADERSCORE^0
junk
}
#
######################################################################
_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail