1. Always check for virus/spam before checking for valid reciepient, or
whether the mailbox is full or some such.
Why? Even if SMTP made it easy to do this (it doesn't), offhand it
seems better to perform the easiest and/or more reliable checks first.
It's very difficult to be sure that a message is spam - the same
message can be spam for one recipient and valid for another. However
the authoritative MX for a domain ought to be able to tell whether an
recipient address from that domain is valid. And to me it makes more
sense to reject a message for a reliable reason (like, this recipient
address is not valid) than for an unreliable reason (like, we think
this is spam).
2. Generate a specific error message, maybe we should introduce a
standard error code for this, like 551 - mail rejected as virus or
spam.
3. If the mail is virus or spam, then do not send it back to the
sender - as this is most likely a forged address anyway. Discard it
instead. But if you must, then use the standard error mesaage as
described above.
I see no need to define a new SMTP reply code - an enhanced status code
should suffice. Especially since, as you point out, when the SMTP
server does believe that the message is spam, bouncing the message is
usually the wrong thing to do anyway.
I think with this scheme, we would have avoided alomost all of the
virus/spam and also annoying error traffic .
No need for new protocols, closed networks etc. Maybe a need for some
RBL listing virus/spam infected machines, I don't know.
Third-party RBLs are a really, really, really bad idea. They should be
categorized as Worst Practices.
There might be some cases where the _authoritative_ DNS zone for a MAIL
FROM domain or the _authoritative_ DNS zone for the source IP address
of an SMTP connection says "messages that look like this are not
authorized". And in some of those cases the SMTP server might be able
to tell that a message "looks like this" from just MAIL FROM and/or
RCPT TO. In those cases it _might_ make sense for the SMTP server to
return 2xx in response to those commands (if they are otherwise valid)
just so that it can black-hole the message. Though I think it might
make even more sense to return 4xx in response to those commands - or
just stop responding to that connection - and then refuse to accept any
new connections from that source IP for some period of time. The
rationale here is that the SMTP server doesn't want to waste bandwidth
or cpu cycles by sucking down a message that's going to be discarded
anyway, and it does want to avoid being interrupted by a compromised
machine or spammer's machine if it tries again later.