Re: [ietf-smtp] Proposal: Updating VERP Specification

Hector Santos writes:

Hi Sam,
There is some discussion in the IETF-SMTP list regarding VERP. I would liketo see it updated, codified, modernized, lessons leaned, but also dealingwith technical length limits. See below for my proposal to be updated. Butplease check out the IETF-SMTP list.
What do you think?

Now that's a nice blast from the past. I don't even remember actuallysubmitting the draft, but I guess I did. My main intent was mostly publicdocumentation of the work that I did at that time (and still do).

I sifted through my archives and found my original working nroff source,which I'm attaching below, if it helps. I have no idea if 'roff is still thesuggested working format. The following version includes a few changes Imade to the working code after the original draft.



#! /bin/sh
nroff -ms <<EOF | perl -e '

while (<>) {                            # Read the entire input file.
   s/FORMFEED(\[Page\s+\d+\])\s+/        \1\n\f\n/g;
   s/\f\n$//;
   print;
}'
..pl 10.0i
..po 0
..ll 7.2i
..lt 7.2i
..nr LL 7.2i
..nr LT 7.2i
..ds RF FORMFEED[Page %]
..ds LF S. Varshavchik
..ds CF Expires XXX XX, XXXX
..ds LH VERP SMTP Extension
..ds RH XXX XX, XXXX
..ds CH S. Varshavchik
..hy 0
..ad l
..in 0
..\".nf
..\"INTERNET-DRAFT                                            S. Varshavchik
..\"Expires XXX XX, XXXX                              Double Precision, Inc.
..\"                                                            XXX XX, XXXX
..\"
..\".ce
..\"Variable Envelope Return Path SMTP Extension
..\".ce
..\"draft-varshavchik-verp-smtpext-04.txt
..\"
..\".ti 0
..\"Status Of This Memo
..\"
..\".fi
..in 3
..\"This document is an Internet-Draft and is in full conformance
..\"with all provisions of Section 10 of RFC2026.
..\"
..\"Internet-Drafts are working documents of the Internet Engineering
..\"Task Force (IETF), its areas, and its working groups.  Note that
..\"other groups may also distribute working documents as
..\"Internet-Drafts.
..\"
..\"Internet-Drafts are draft documents valid for a maximum of six
..\"months and may be updated, replaced, or obsoleted by other
..\"documents at any time.  It is inappropriate to use Internet-
..\"Drafts as reference material or to cite them other than as
..\""work in progress."
..\"
..\"The list of current Internet-Drafts can be accessed at
..\"
..\"http://www.ietf.org/ietf/1id-abstracts.txt
..\"
..\"The list of Internet-Draft Shadow Directories can be accessed at
..\"http://www.ietf.org/shadow.html.
..\"
..\".ti 0
..\"0. Revision history
..\"
..\"01 - added additional comments on DSNs not being very successful in
..\"automatically handling mailing list functions.  Additional section
..\"added with comments regarding vacation autoresponders.
..\"
..\"02 - expanded section 6 to describe the proper way to handle local
..\"aliasing, or address explosion.
..\"
..\"03 - added the characters -, [, and ], to the list of characters that
..\"must be hex-encoded in a VERP, as well as requiring the domain portion
..\"of the return address to be encoded.  This resolves certain ambiguities in
..\"potential return addresses, as well as allow network address literals
..\"to be used.  Adjusted example SMTP conversation to reflect proper reply
..\"codes.
..\"
..\"04 - clarify that upper and lowercase hexadecimal characters are permitted.
..\"Update contact information.
..\"
..ti 0
1. Abstract

This document describes an extension to the SMTP service [1],
called Variable Envelope Return Path (VERP).
The VERP extension implements a way of automatically identifying
undeliverable mail recipients, even
when non-delivery reports originate from mail systems that do
not implement delivery status notifications, as specified in [2] and [3].

..ti 0
2. Introduction

All E-mail software can expect to deal with undeliverable mail.
[2] and [3] specify a machine-readable format for delivery status
notifications (DSNs, or non-delivery reports).
DSNs allow
undeliverable mail to be handled in a totally automatic fashion, without
requiring manual intervention.
For example, mailing list managers
can automatically identify addresses that are no longer deliverable,
and remove them from the mailing list.

Although [2] and [3] are now widely implemented,
there are still many systems
that do not use them.
This makes it impractical to completely rely on DSNs for
automatic mailing list management.
Undeliverable addresses accumulate quickly even from
a very small percentage of non-DSN systems.
This results in a non-trivial amount of manual work to identify undeliverable
addresses and purge them from the mailing list.

Mailing list software began to use VERPs
(the acronym stands for Variable Envelope Return Path)
after DSNs were found to be impractical for totally
automatic mailing list management.
VERPs are an alternative way to handle non-delivery notices.
The advantage of VERPs is that they can be made to work
automatically, even when non-delivery
notices are not in the format specified by [2].

Unfortunately, VERPs require much more bandwidth and network resources
than DSNs because VERPs cannot be used to send one copy of a
mailing list message addressed to all the recipients in the same E-mail domain.

This SMTP service extension allows E-mail software to send a single
VERP message to all addresses in the same mail domain,
for as long as mail servers, which relay the message, support the VERP
SMTP extension.

The VERP message may be eventually relayed to a mail server that does not
support this extension.
Separate messages -
with variable envelope return paths -
will be sent when this happens.

So the worst case scenario results in the same situation where
traditional VERPs are used right from the start.
The best case scenario results in significant savings of
network resources and bandwidth, from eliminating hundreds (or
more) copies of the same message.

Essentially, the VERP extension postpones the generation of multiple
messages with different return paths as much as possible, until
it is absolutely required.

..ti 0
2.1 VERP overview

The traditional VERP message encodes
the recipient address as a portion of the return address.
When undeliverable mail comes back, the mail software
decodes the return address (now the recipient address) and
obtains the address responsible for the non-delivery notice.

For example: mail sent by a mailing list
manager to the address <john(_at_)example(_dot_)org>
carries a return address of 
<mlist-return-john=example(_dot_)org(_at_)domain(_dot_)com>.
The mailing list software at domain.com handles all mail with the local
portion of the address starting with "mlist-return-".
If a non-delivery notice is generated because the address is not deliverable,
the mailing list software
takes the address where the non-delivery report was sent, retrieves the
remaining portion of the local address, "john=example.org", and determines
that the undeliverable address was <john(_at_)example(_dot_)org>.

This does not rely on RFC 1894, and will work for all non-delivery notices.

..ti 0
3. Framework for the VERP SMTP transport extension

This SMTP transport extension [1] is laid out as follows.

..in +7
..ti -4
(1) The name of the SMTP transport extension defined here is Variable
Envelope Return Path.

..ti -4
(2) The EHLO keyword associated with this extension is VERP.

..ti -4
(3) The VERP EHLO keyword takes no parameters.

..ti -4
(4) One optional ESMTP keyword VERP is associated with the
MAIL FROM command. This parameter takes no values.

..ti -4
(5) No additional ESMTP verbs are defined by this extension.

..ti -4
(6) The next section specifies how support for this extension affects
the behavior of a server and client SMTP.
..in -7
..\"The IANA registry is ammended as follows.  "ESMTP MAIL KEYWORDS"
..\"are ammended as follows:
..\"
..\".nf
..\"Keywords Description                    Reference
..\"-------- -----------------------------  --------------------------------
..\"VERP     Variable Envelope Return Path  [draft-varshavchik-verp-smtpext]
..\".fi
..\"
..\""MAIL EXTENSION TYPES" are ammended as follows:
..\"
..\".nf
..\"Service Ext   EHLO Keyword Parameters Verb   Reference
..\"-----------   ------------ ---------- ------ 
--------------------------------
..\"Var Env RP       EHLO         none     none
..\".fi

..ti 0
4. The VERP SMTP extension

When a VERP keyword is present in the MAIL FROM command, [4],
some additional restrictions are imposed
on the RFC 822 address [5], specified by that
MAIL FROM
command, and on
all RFC 822 addresses in the subsequent RCPT TO
commands that refer to the same message (that is, until the next DATA,
RSET, or QUIT command).
The term "VERP message" refers
to any E-mail message whose MAIL FROM command includes the VERP keyword.
The term "VERP-compliant server" refers to any
E-mail server that supports the Variable Envelope Return Path SMTP
extension.
When a VERP keyword is present in the MAIL FROM command:

..in +8
..ti -4
(1) The address specified by the MAIL FROM verb MUST contain
at least one @ character.

..ti -4
(2) The address in every RCPT TO verb referring to the same
message MUST contain at least one @ character.

..ti -4
(3) The domain portion of the address
in the MAIL FROM and RCPT TO verbs
MUST be compliant with the definition of <domain>
in [6].  That is, it MUST contain only letters, digits, hyphens, and
periods.
The domain portion of the address is the one
that follows
the last @ character.
..in -8

..ti 0
4.1 Delivery failures

When a VERP-compliant
server is unable to deliver a VERP message to one or more
recipients, the VERP server MUST do one of the following:

..in +8
..ti -3
1) Return an RFC 1891 delivery status notification to the
return address, or:

..ti -3
2) Transmit a separate non-delivery notice for each failed recipient.
The return address for each non-delivery notice MUST be
the address that's formed by applying the procedure
described in section 7 of this
document to the return address of the message and the
failed recipient's address.  If more than one recipient was undeliverable
a separate notice MUST be sent for each undeliverable address.
..in -8

..ti 0
5. Final delivery

Section 4.3.1 of [5] specifies that the mail server performing final
delivery of a message will generate a Return-Path: header containing
the return address of the message.

This return address MUST be formed by applying the procedure
described in section 7 of this document to the return address and
the recipient's address.

This also applies if the mail server invokes some other external
process to handle final delivery, instead of placing the message into
the recipient's mailbox. In all cases, the return address specified
by the mail server to any external environment or process MUST be
derived by applying the procedure in section 7 to the return address
and the recipient's address.

..ti 0
6. Relaying

When a VERP-compliant server determines that a recipient
of a VERP message is not a local mailbox, and the message
must be relayed to another server, the VERP-compliant server MUST:

..in +8
..ti -4
(1) If the VERP-compliant server's local policies require
the return and/or recipient addresses are to be rewritten,
the VERP-compliant server MUST make sure that delivery notices
MUST NOT be sent to a rewritten return address, or reflect
a rewritten recipient address, unless separate arrangements are
made with the sender of the VERP message.
This is because the sender expects to be able to
resolve non-delivery notices to some recipient address it knows
about (irrespective whether the addresses are resolved from a VERP,
or from a delivery status notification in the format specified by [2]).
When local policies of the relaying server require the
recipient or the sender address
to be rewritten,
the sender will obviously be not aware of this rewriting or expansion.
If a
subsequent delivery failure results in a non-delivery notice being sent
to a VERP containing the rewritten address, the sender will not be able
to associate the address encoded in the VERP with any recipient address
it has on file.

A suggested way to handle this situation is to set specify a delivery
status notification of "NEVER" - as specified by [3] -
for any recipient address generated by local aliasing or
rewriting, and treat the original address as being
either delivered or relayed, resulting in a
"delivered" or "relayed" delivery status notification if the original
recipient address specified a "SUCCESS" notification.

Note that this clause does not prohibit the relaying server to
have an established agreement with the sender to act as a mailing list
exploder, as long as this agreement is handled in a transparent way.
For example, the relaying server can set its own return address on all
VERP messages it exploded, and handle non-delivery notices all by itself.

..ti -4
(2) If the VERP-compliant server determines that the remote server is
also a VERP compliant server, the VERP keyword MUST be included in the
MAIL FROM command used to relay the VERP message to the remote server.

..ti -4
(3) If the remote server is not a VERP compliant server,
The VERP compliant server SHOULD
send a separate copy of the message for every recipient.
The return address of each
copy of the message MUST be formed by applying the procedure
described in section 7 of this document to the original return address,
and the address of each individual recipient.
Although the message SHOULD NOT be returned as undeliverable,
if it is then the rules defined in section 4.1 MUST be applied.

These rules also apply
if the SMTP-compliant server determines that the VERP message must be
forwarded via some other protocol to a non-SMTP gateway, unless the
non-SMTP
protocol has equivalent features that are completely identical in
function to Variable Envelope Return Path SMTP service extension
(including
any translations of E-mail addresses to and from the non-RFC822 format).
..in -8

..ti 0
7. Variable envelope return path encoding

This encoding method starts with a return address and one recipient
address.  As
mentioned previously, both addresses MUST be valid RFC822 addresses, [5],
and MUST
contain at least one @ character.
The portion of each address following the last @ character MUST
be compliant with [6], or must be an explict network address literal,
for example: john43@[192.68.0.4].

Let "sdomain" represent the portion of the return address that
follows the last @ character.

Let "slocal" represent the portion of the return address that
precedes the last @ character.

Let "rdomain" represent the portion of the recipient address that follows
the last @ character.

Let "rlocal" represent the portion of the recipient address that precedes
the last @ character.

To encode the recipient address within the envelope sender address,
create an address of the following form:

..ti +4
  slocal-encodedrlocal=encodedrdomain@sdomain

Where "encodedrlocal" and "encodedrdomain" are formed by taking
rlocal and rdomain, respectively, and encoding each one as follows:

..in +8
..ti -3
1) Each @, :, %, !, -, [, ], and + character in rlocal
is replaced by a single '+' character
followed by two uppercase hexadecimal characters whose value is the ASCII
code of the replaced character.

Taking into account poorly-written mail relays that ignore case-sensitivity,
both uppercase and lowercase hexadecimal characters SHOULD be recognized
when decoding envelope return paths.

..ti -3
2) All other characters are unchanged.  Other characters MAY, but SHOULD NOT
be also encoded in the same fashion.
..in -8

This can be represented using BNF as follows:

..in +2
..nf
encodedverp: slocal "-" encodedrlocal "=" encodedrdomain "@" sdomain

encodedrlocal: * (char-literal / char-encoded )

encodedrdomain: * (char-literal / char-encoded )

char-literal: any character valid in an RFC821 address [4],
             except @, :, %, !, [, ], -, and +

char-encoded: "+" hexdigit hexdigit

hexdigit: ("0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" /
          "9" / "A" / "B" / "C" / "D" / "E" / "F" /
                "a" / "b" / "c" / "d" / "e" / "f")
..fi
..in -2

..ti 0
8. Variable envelope return path decoding

Non-delivery notices for VERP messages
will be sent to either the original address, <slocal@sdomain>, or to the
VERP-encoded address, <slocal-encodedrlocal=encodedrdomain@sdomain>.

Messages sent to <slocal@sdomain> will be RFC 1891-compliant delivery
status notifications.  These messages will be machine-readable, and the
mail software will be able to
identify failed addresses from the RFC 1891 delivery report.

Non-delivery notices will also be sent to
the VERP-encoded address, and the mail software will be able to reconstruct
the failed address from the VERP-encoded address
by simply reversing the steps used in encoding:

..in +8
..ti -3
1) Extract encodedrlocal and rdomain from the recipient address.
There will be at least one = character in the encoded portion of the
return address.  encodedrlocal is everything up to the last =
character.
Everything following the last = character is encodedrdomain.

..ti -3
2) Replacing all occurrences of "+" followed by two hexadecimal digits
in encodedrlocal and encodedrdomain with the equivalent ASCII character.

..ti -3
3) Using the decoded rlocal, @, then rdomain.
..in -8

..ti 0
9. Examples

Suppose that a VERP-compliant server named "example.com" receives a message
via the following SMTP conversation (for brevity, non-relevant headers have
been omitted):

..in +4
..nf
220 example.com ESMTP
EHLO domain.com
250-example.com ESMTP
250-SIZE
250-DSN
250-VERP
250 HELP
MAIL FROM:<itny-out(_at_)domain(_dot_)com> VERP SIZE=100
250 Ok
RCPT TO:<alex(_at_)example(_dot_)com>
250 Ok
RCPT TO:<node42!ann(_at_)old(_dot_)example(_dot_)com>
250 Ok
RCPT TO:<tom(_at_)old(_dot_)example(_dot_)com>
250 Ok
RCPT TO:<lisa(_at_)new(_dot_)example(_dot_)com>
250 Ok
RCPT TO:<dave+priority(_at_)new(_dot_)example(_dot_)com>
250 Ok
DATA
354 Ok
From: "John" <john(_at_)domain(_dot_)com>
Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST)
Subject: Meeting canceled.

Today's 2pm meeting has been rescheduled for tomorrow, 9am, due
to a scheduling conflict.
\&.
..fi
..in -4

The message is delivered to the local mailbox for <alex(_at_)example(_dot_)com>.
The message looks like this:

..in +4
..nf
Return-Path: <itny-out-alex=example(_dot_)com(_at_)domain(_dot_)com>
From: "John" <john(_at_)domain(_dot_)com>
Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST)
Subject: Meeting canceled.

Today's 2pm meeting has been rescheduled for tomorrow, 9am, due
to a scheduling conflict.
..fi
..in -4

The VERP-compliant server at example.com connects to the mail server
for old.example.com.
old.example.com does not support the
Variable Envelope Return Path extension.
Therefore, old.example.com receives two messages.  The SMTP
conversation for the first message is as follows:

..in +4
..nf
220 old.example.com ESMTP
EHLO example.com
250-old.example.com ESMTP
250-SIZE
250-DSN
250 HELP
MAIL 
FROM:<itny-out-node42+21ann=old(_dot_)example(_dot_)com(_at_)domain(_dot_)com>
250 Ok
RCPT TO:<node42!ann(_at_)old(_dot_)example(_dot_)com>
250 Ok
DATA
354 Ok
From: "John" <john(_at_)domain(_dot_)com>
Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST)
Subject: Meeting canceled.

Today's 2pm meeting has been rescheduled for tomorrow, 9am, due
to a scheduling conflict.
\&.
..fi
..in -4

The SMTP conversation for the second message is as follows:

..in +4
..nf
MAIL FROM:<itny-out-tom=old(_dot_)example(_dot_)com(_at_)domain(_dot_)com>
250 Ok
RCPT TO:<tom(_at_)old(_dot_)example(_dot_)com>
250 Ok
DATA
354 Ok
From: "John" <john(_at_)domain(_dot_)com>
Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST)
Subject: Meeting canceled.

Today's 2pm meeting has been rescheduled for tomorrow, 9am, due
to a scheduling conflict.
\&.
..fi
..in -4

example.com connects to new.example.com and determines that
new.example.com runs a modern ESMTP server that supports the VERP
keyword. The SMTP conversation then goes like this:

..in +4
..nf
220 new.example.com ESMTP
EHLO example.com
250-new.example.com ESMTP
250-SIZE
250-DSN
250-VERP
250 HELP
MAIL FROM:<itny-out(_at_)domain(_dot_)com> VERP SIZE=100
250 Ok
RCPT TO:<lisa(_at_)new(_dot_)example(_dot_)com>
250 Ok
RCPT TO:<dave+priority(_at_)new(_dot_)example(_dot_)com>
250 Ok
DATA
354 Ok
From: "John" <john(_at_)domain(_dot_)com>
Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST)
Subject: Meeting canceled.

Today's 2pm meeting has been rescheduled for tomorrow, 9am, due
to a scheduling conflict.
\&.
..fi
..in -4

..ti 0
10. Security concerns

All the usual security considerations applicable to SMTP are also
applicable to this extension.
Relay of VERP messages to non-VERP servers requires a single
message with many recipients
to be exploded into many messages with
one recipient.
In all cases, however, there will never be any additional overhead
beyond the resources that are required when VERPs
are manually implemented by the mail sender, instead of
the VERP SMTP extension.

Mail systems which support the VERP extension SHOULD have adequate
security measures, including blocks against unauthorized access and
relaying.

..ti 0
10.1 Vacation programs, and other autoresponders

"Vacation" type autoresponders are often used in practice.
A vacation autoresponder is a program that automatically replies to
every message, informing the sender that the
recipient is on vacation, or is generally unavailable at this time.

Vacation autoresponders MUST NOT generate autoresponses to mailing
list messages, but people often forget to do set them up to do so.
Because autoresponses are sent to the same address that's used to
receive non-delivery reports, malfunctioning autoresponders result
in the recipient being removed from mailing lists.

Advanced autoresponders send automatic replies in the format specified
by [2], as a "delayed" notification.  DSN-aware software will not
remove addresses from mailing lists due to delayed notifications.

Section 5 of this document specifies that the mail server MUST
replace the original return address with a VERP-modified address
when delivering the message to a mailbox or an external process.

Therefore it is possible
that RFC 1891 reports may also be sent to
a VERP-encoded address, as specified by sections 5 and 7 of this document.
Mail software SHOULD ignore any RFC 1891 "delayed" or "success" reports that
sent to a VERP-encoded address.
If it is a "failed" report, note that the VERP address will be
more reliable than the address specified in the report itself.

..bp
..ti 0
11. References

..in +8
..ti -4
[1] Klensin, J., Freed, N., Rose, M., Stefferud, E., Crocker, D.
"SMTP Service Extensions", RFC 1425, United Nations University,
Innosoft International, Inc., Dover Beach Consulting, Inc.,
Network Management Associates, Inc., The Branch Office,
February 1993

..ti -4
[2] Moore, K., and G. Vaudreuil, "An Extensible Message Format for
Delivery Status Notifications", RFC 1894, University of Tennessee,
Octel Network Services, January 1996.

..ti -4
[3] Moore, K. "SMTP Service Extension for Delivery Status
Notifications", RFC 1891, University of Tennessee, January 1996.

..ti -4
[4] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821,
USC/Information Sciences Institute, August 1982.

..ti -4
[5] Crocker, D., "Standard for the Format of ARPA Internet Text
Messages", STD 11, RFC 822, UDEL, August 1982.

..ti -4
[6] Mockapetris, P., "Domain Names - Implementation and
Specification", RFC 1035, ISI, November 1987

..in -8
..ti 0
12. Author's address

..nf
Sam Varshavchik
Double Precision, Inc.
402 Main Street Suite 100-241
Metuchen, NJ 08840
<mrsam(_at_)courier-mta(_dot_)com>
..fi
EOF

pgpqGFiAioWkp.pgp
Description: PGP signature

_______________________________________________
ietf-smtp mailing list
ietf-smtp(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf-smtp