[Top] [All Lists]

Re: [Asrg] 6. Proposals - A Brief Defence of "Pull"

2003-10-01 13:07:53
At 2:07 AM +1000 2003/10/02, Brett Watson wrote:

 A large volume of spam is currently shifted through proxy servers: either
 misconfigured HTTP proxies, or virus-compromised PCs with added malware
 (zombies) [ref.]. They provide
 a convenient way for a spammer to hide behind someone else's IP address.

        With you so far.

 Requiring a callback to fetch the message data makes this approach much
 harder for a spammer, as it would require proxies in both directions: an HTTP
 proxy will not suffice at all, and the longer the callback is delayed, the
 more likely a zombie will have gone offline (at least temporarily), thus
 causing delivery failure.

I disagree. In the case of zombies, which is becoming more and more the method of choice, they have complete control of the machine. They can send out the notices, wait for the callbacks, and then send the payload. All safe in the knowledge that they control the machine. Moreover, a network of zombies could easily be configured so that one zombie designates another as the call back server, and that process could even be distributed and load-balanced.

Consider viruses which propagate by means of their own SMTP engine: by far the
 most common variety, as I understand. They distribute themselves
 opportunistically, relying on immediate delivery; if they encounter a
 temporary failure, they just move on to the next target rather than wasting

Many spammers and viruses are already doing two rounds of attempts on the same recipient addresses. They don't actually queue, per se, in that they have only one copy of the message body and they do a mail-merge to the massive group of recipients (allowing them to operate entirely out of memory for maximum speed), but they do track status for the recipients and retry failures.

        Many spamming tools seem to work the same way: spammers compensate for
 delivery failures by attempting delivery to a larger number of addresses,
 rather than sitting around and waiting for the mail to get through to
 unresponsive sites. This is why greylisting works [ref.].

See above. Greylisting is currently working for a certain percentage of messages, but that percentage is rapidly decreasing. It will continue to be a valid useful tool in the future, but only in conjunction with other tools, especially message content scanning/central registry tools such as DCC/Vipul's Razor/Pyzor, and realtime per-IP address spam-reporting/blacklist tools. Greylisting gives them a much better chance of being able to catch and quarantine messages based on reports from other people.

The next phase will be to rapidly change the message bodies fast enough so that DCC/Razor/Pyzor become less and less useful. Same for realtime IP address reporting/blacklist tools.

The "pull" mode of delivery complements this very well, because in a pull-mode
 delivery, the receiving system controls the timing of delivery. Greylisting
 is like saying, "if you're serious about sending me this message, call me
 back later." Pull-mode delivery is like saying, "if you're serious about
 sending me this message, let *me* call *you* back later, when *I* feel like

That only affects the overall issue when done end-to-end. If you try to slot this in at the link layer, it doesn't help.

If telemarketers have to call a number and leave a message for a callback but you are not in direct control of that callback, you are no better off than if they had called you directly.

Indeed, but you are under modestly significant timeout constraints if you want
 to pause to check various things in the middle of an SMTP dialogue. With
 pull, you can accept the envelope data, then do your policy checks on it in
 your own time.

Envelope checks happen quickly anyway. If you're getting so many of them that you bog down doing them interactively, then you're not likely to have your situation improved by accepting everything and then trying to do them in the background. If anything, you're likely to be put in a worse position, as you get overrun by spam and viruses and you can't properly deal with the real messages.

With both spam and viruses, you're best off if you can detect them interactively and refuse to accept them in the first place. If the sender manages to successfully transmit the message to you without being detected, they've won.

                 Timeout constraints are relaxed by at least an order of
 magnitude, and you're not holding a TCP connection open while you do it,
 either. So there is a benefit here, albeit a relatively minor one.

If you want to do background mode for envelope checks, it could always be a fallback from interactive mode, although I still believe that it's a bad idea.

The "message pull" storage server is an outgoing MTA -- one that normally acts
 in a role of accepting SMTP connections from your network and doing a
 store-and-forward to the remainder of the Internet. It may also be an
 incoming MTA. If spammers have compromised it, you have *much* bigger
 problems than the scenario you mentioned, regardless of your mail transfer

Yeah, but you've also given them a much bigger target. Why do criminals still go after banks, even with all the security, guards with guns, and everything else? Because that's where the money is.

You're concentrating by many, many orders of magnitude the attractiveness of the target. You're also concentrating by many orders of magnitude the amount of reliability that has to be provided by that server. It is not only responsible for storing messages temporarily while they are being transmitted, now it also has to hold all those messages while they're waiting for callback.

Moreover, for spammers or viruses, for the callback they can always designate another server they control, or themselves.

 I still don't see that this is a realistic threat, but as I said, the sending
 MTA can pass a SHA1 digest with the envelope data, and thus commit to a
 message in exactly the manner you desire. Or would that still be
 unsatisfactory for some reason?

It's yet another change to the protocol that has to happen, because you're trying to change the fundamental nature of the way e-mail works. It's a patch to an already bad patch, and that doesn't improve the situation.

 A SHA1 digest would provide an additional DCC-like data point to evaluate as
 part of the policy check, and I can see some value in that. The sender could
 also commit up-front to a message size, and that would be useful. Both of
 those data points can be verified after download, and the message rejected
 due to "data corruption" if the checks fail. These are additional ideas which
 can be sensibly built onto the pull-mode base if they are considered

They would be necessary, but you would need to make them even more complex because you'd have to deal with things like changes in the content-transfer-encoding which change the internal representation of the message, but not the actual content of the message itself.

This is the same sort of problem that PGP-signed messages have today. Think about how many mailing lists there are which still break PGP signatures, not to mention less intelligent MTAs, etc....

 From this point on, I'm afraid that you lost me again. I think you're arguing
 against a scheme that someone else proposed. I'm attempting to argue that the
 smallest possible addition of pull-mode behaviour into SMTP would make it
 more robust against many mail-abuse techniques:

I don't think you've successfully shown that. Many times there are methods which are suitable at the link layer that are not suitable for end-to-end, or vice-versa. Using a pull-based mechanism has been demonstrated to work for USENET and web discussion groups.

But you have yet to prove that you can successfully use this end-to-end mechanism to any benefit for the link-layer (especially since most e-mail is point-to-point today anyway), or that you can successfully turn all Internet e-mail into USENET news or web discussion groups.

Brad Knowles, <brad(_dot_)knowles(_at_)skynet(_dot_)be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
    -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

Asrg mailing list