Hector Santos wrote:
Yakov,
I am seeing things or what?
We have the CallerID Verification system now in place for at least 4-5 days.
This method has drastically reduced the spam via unverified return paths,
but practically 100%.
The method you are refering is similar to what is more commonly known as
"Challenge/Response" (C/R). Majority of C/R implementation actually send
a message back to the sender asking him/her to click on a link to reply
to the message in order to verify that the sender exists. Some go
further and use reverse-Turing tests with images that supposedly only
humans and not machines can read.
Your particular method does not actually send the email to the sender,
but stops as soon as the RCPT TO: path has been found to be valid which
varies from the C/R systems.
Am I seeing things? Why isn't the group homing in on this method which
basically says that the return path must be valid? I read the pros and
cons in the PDF you provided to me. It didn't make sense to me (the CONS).
The simple reason why C/R works is because spammers don't care. At this
point a very small minority of email systems in the world use any kind
of C/R, and therefore spammers do not bother spending the time to fight
them. Some C/R companies such as MailBlocks, etc. report virtually 99%
success rate. However, if C/R usage spreads, then spammers will have an
incentive to attack them, and they will. Also, spammers tend to share
their results much easier then some of the anti-spam people.
There are problems with the the regular C/R approaches that apply here
as well:
1. Nothing stops a spammer from using someone else's email address with
that domain answering yes (unless you start matching the his email
address with his IP which is what the LMAP proposals do). Especially
this would be true if he uses YOUR domain. Also, as per RFC 2822 you
must support MAIL FROM: <> which basically allows spammers to bypass
everything at this step.
2. Even checking MAIL FROM does not prevent spammers from forging the
"From" header inside the actual email message.
While there are numerous problems with regular C/R, your particular
approach has some issues as well:
1. Nothing stops a spammer from using his own domain which answers "yes"
to all RCPT TOs to that domain (BUT not an open relay).
2. ISPs tend to play around with RCPT TO: handling such as answering
"yes" to all requests, or delaying responses in order to make sure
spammers do not use RCPT TO: for harvesting email addresses.
3. Since your proposal operates at the MAIL FROM step, if the sender's
receiving MTA is off-line this process will fail. This will also be a
problem if the sender is using GreyListing
(http://projects.puremagic.com/greylisting/). According to RFC 2821,
section 4.5.4.1., the originator must be able to resend email several
times when temporary failure occurs, which in practice happens sometimes.
As I noted to you via EMAIL, our system allows for programmable hooks at
each step of the protocol state machine:
connect --> connect IP validate hook
Would this be an rDNS / RBL hook?
HELO/EHLO ---> helo/ehlo validate hook
What kind of hook are you using here? If a spammer uses a valid domain
here, nothing stops him. Only something like LMAP/DRIP that ties IP
addresses with domain names for SMTP can help against that.
MAIL FROM: ---> mail from validate hook
What kind of hook aside from this proposal are you using here?
RCPT TO: ---> recipient validation hook
DATA: ---> mail filter validation hook (3rd party software hooks)
We offer internal support for RBL at the connect level, restrict conformity
to HELO/ECHO, local user validation at RCPT TO: and 3rd party hooks for the
DATA level. Now we added WCSAP (Wildcat! Sender Authentication Protocol)
to the MAIL FROM: state where the callerid verification takes place.
I added additional logic to make it even stronger.
If the CallerID verification process receives a 250 positive response in the
RCPT TO: return path validation, the logic then goes a step further to check
if the site is an open relay (see if will it accept any random address).
What happens with 4xx codes? or if the MTA is off-line? In regular C/R,
the challenge message will be resent multiple times as per RFC 2821,
section 4.5.4.1 an that avoids the problems mentioned above.
[...]
Also, I read little about JAMSPAM.
JamSpam seems to be currently dead. The folks involved could not agree
on anything so it fell apart.
You probably discussed it already, but the technical standards should
indicate and emphasis restrictions to the transports protocol is purely
technical. EMAIL SYSTEMS invalid handshaking is all the IETF should be
looking at to simplified this process. To go any further, simply will
complicate things and give JAMSPAM proponents a reason to exist. Their
layman argument is purely based on the legacy operations of the SMTP system.
Imposed stronger state point handshaking and these people have no basis for
argument other than "social."
What kind of controls are we talking about? Something like LMAP?
Unlike others I read in the group, this is a TECHNICAL problem. Not a social
issue.
The idea of "filtering the data" once the transport system has done its job,
is purely a system level admistrative option and/or feature. This is akin
to the FTP protocol. In our system, we offer a "Scan File" option once a
file is uploaded, but this is an admin/customer issue. Has nothing to do
with the protocol itself. The technical process must adhere to standards
otherwise it (the session) is rejected.
RFC 2821 states (sections 3.3, 4.1.1.4 and 6.1) that the SMTP server
has two options: accept or reject the message, and has two options to
send back OK or Failure (5xx). HOWEVER, once the message has been
accepted as per this section (6.1):
" When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
message in response to DATA), it is accepting responsibility for
delivering or relaying the message. It must take this responsibility
seriously. It MUST NOT lose the message for frivolous reasons, such
as because the host later crashes or because of a predictable
resource shortage.
If there is a delivery failure after acceptance of a message, the
receiver-SMTP MUST formulate and mail a notification message. This
notification MUST be sent using a null ("<>") reverse path in the
envelope. The recipient of this notification MUST be the address
from the envelope return path (or the Return-Path: line). However,
if this address is null ("<>"), the receiver-SMTP MUST NOT send a
notification. Obviously, nothing in this section can or should
prohibit local decisions (i.e., as part of the same system
environment as the receiver-SMTP) to log or otherwise transmit
information about null address events locally if that is desired. If
the address is an explicit source route, it MUST be stripped down to
its final hop.
"
Therefore, the RFC does dictate what should been done with the message
once it has been filtered. Even though many anti-spam systems honor this
"in breach", this is another illustration of how protocols affect other
things, even filtering.
You want things to get moving, get the proposals that concentrate on the
technical aspects of SMTP who do not have any association whatsoever with
mail content or links to "databases" that could make for questionable
non-technical restrictions. That shouldn't be an SMTP issue.
The bottom line is that once a strict SMTP protocol is put in place, the
SPAMMERS will have no choice but to comply because those are the secured
technical rules. In my view, this will clean up the spammer problem in
short order. They must comply from a technical standpoint. Period. It has
nothing to with SPAM, really.
It will only work if most of the mail systems on the Internet comply
with it. The IETF cannot force compliance with its standards. Therefore,
partly what we are seeking is an incremental solution as stated in the
technical considerations document
(http://www.ietf.org/internet-drafts/draft-crocker-spam-techconsider-02.txt):
" By contrast, some "edge" mechanisms provide utility to
the first one, two or three adopters who interact with
each other. No one else is needed for the adopters to
gain some benefit. Each additional adopter makes the
total system incrementally more useful. For example a
filter can be useful to the first recipient to adopt
it. A consent mechanism can be useful to the first two
or three adopters, depending upon the design of the
mechanism.
"
We are open to hearing how the existing SMTP protocol may be tightened
but its must be proven that this will solve the problem.
In any case, we are going to finally WCSAP CallerId Verification in our
Wildcat! mail system and go gamma with it next week to thousands of
customers. Ironically, I already even have received a concern from a 3rd
party Anti-Spam author saying "This is going to hurt our sales!" My
response was simple: "non-conforming anonymous systems into the mail
transport is now going to be a thing of the past. Mail from conforming
systems will be allowed to pass thru which is where your product comes into
play with its mail filtering techniques optionally offer to the system admin
and for users of the system."
I am sure that you will allow for more than "4-5 days of testing" before
deploying this.
The thing is, unless I am seeing things, with WCSAP, we are blocking nearly
100% of the darn spam! I have carefully looked at the looks for false
positives. None!
IF you are rejecting email at the MAIL FROM stage, there is now way to
check for false positives since you do not have the actual email
message. Additionally, your specific test installation might not be
large enough to extrapolate to the entire Internet. If you can provide
additional statistical information, that would be useful.
I'm sure that spammers will eventually "conform" to technical standards. In
my view, from a SMTP standpoint, that is all the IETF and this working group
should care about.
Spammers do not care - they will only conform to standards if they are
forced to do so in order to deliver their garbage. Therefore, in order
for any standard to work, existing email systems on the Internet must be
willing to adapt them.
As a mail processor/gateway developer for several networks over the last 25
years, I am firm believer in unaltered passthru of mail. In my view, it is
unethical to a) restrict mail reaching its destination and b) modify it in
any way. However, that has nothing to do with technical conformity. The
handshaking process should be an identified process on both ends.
On a similar note, the RMX/DNS solutions, etc, are essentially the same
idea, but it requires DNS administration which in my view is going to be
problematic in all aspects, technically, politically, including
capitalistically (or communistically depending on your point of view).
Because of this, it will be slow to catch on.
Nevertheless, any existing methods do not provide a way during
handshaking to tie in IP addresses to domain names for SMTP. This is
something that must be introduced in order to verify senders, or some
other method.
However, one way to help the process is to include as part of the SMTP
specification, a DNS registration process. This will force (encourage)
developers to include a "registration" process as part of their SMTP
software installation or setup process. This registration process can
include other attributes, besides RMX related, such as "type of system." Is
it open, closed (private) system? Does the system offer open spam,
restricted spam, or no spam? etc.
Of course, this extended attribute concept be implemented using EHLO,
however, if we are going to require a modified MX DNS system, then we might
as well add other attributes to it as well. A conforming client can use
this information and help save at least 1/2 of the bandwidth. A client can
also instantly inform a user or sender that the communication with a target
system will be restricted in one form or another.
Nonetheless, compared to a CallerID Verification concept, the extended DNS
based solutions is going to be a slow and costly endeavor. In my view,
Return Path callerid ID handshaking seems to be a very logical extension to
the SMTP client/server handshaking process. In the current form of SMTP, it
works fantastic. If we anticipate this is going to be a widely accepted
practice for all developers (big, including small, a major plus reason why
it will be accepted faster than a DNS based solution), then maybe we should
look at reducing handshaking requirements to accomplish a callerid
verification SMTP logic.
The following sections from the technical considerations document are
relevant:
" The idea of replacing SMTP is appealing because it
permits thinking in terms of creating an infrastructure
that has accountability and restrictions built in.
Unfortunately an installed base the size of the
Internet is not likely to make such a change anytime
soon. It seems far more likely that successful spam
control mechanisms will be introduced as increments to
the existing Internet mail service.
"
This statement would apply to CHANGING the SMTP protocol as well. Any
major changes would have the same deployment issues. See also the following:
"4.1. Adoption
A critical barrier to the success of a new mechanism is
the effort it takes to begin using it. It is essential
to look carefully at the adoption process.
1) Adoption What is the effort for a new
Effort participant to start using the
proposed mechanism? This includes
installation, learning to use it
and performing initial
operations. This is also called
the "barrier to entry".
2) Threshold What is required before users get
to benefit some benefit from the mechanism?
Primarily, this looks for the
number of users who must adopt
the mechanism before the adopters
gain utility from it.
A key construct to examination of adoption and benefit
is "core-vs-edge". Generally, adoption at the edge of
a system is easier and quicker than adoption in the
core. If a mechanism affects the core (infrastructure)
then it usually must be adopted by most or all of the
infrastructure before it provides meaningful utility.
In something the scale of the Internet, it can take
decades to reach that level of adoption, if it ever
does.
Remember that the Internet comprises a massive number
of independent administrations, each with their own
politics and funding. What is important and feasible to
one might be neither to another. If the latter
administration is in the handling path for a message,
then it will not have implemented the necessary control
mechanism. Worse, it well might not be possible to
change this. For example a proposal that requires a
brand new mail service is not likely to gain much
traction.
By contrast, some "edge" mechanisms provide utility to
the first one, two or three adopters who interact with
each other. No one else is needed for the adopters to
gain some benefit. Each additional adopter makes the
total system incrementally more useful. For example a
filter can be useful to the first recipient to adopt
it. A consent mechanism can be useful to the first two
or three adopters, depending upon the design of the
mechanism.
3) Impact on What is the impact on the senders
Participants and receivers who adopt the
proposal? Senders and receivers
currently have certain styles of
operation. How are those styles
changed?
4) Impact on What is the impact on the senders
Others and receivers who do *not* adopt
the proposal? What effect does it
have on legitimate users of
email? What effect does it have
on spammers? Is the nature of
Internet mail changed for
everyone, including non-
adopters?
For example, a challenge-response system is irritating
for the person being challenged, and it imposes extra
delay on the desired communication. If the originator
and the recipient both access the Internet only
occasionally (such as through dial-up when mobile) a
challenge-response model can impose days of delay. For
some communications, this can be disastrous.
"
-------
Yakov Shafranovich / asrg <at> shaftek.org
SolidMatrix Technologies, Inc. / research <at> solidmatrix.com
"And this too shall come to pass"
-------
_______________________________________________
Asrg mailing list
Asrg(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/asrg