----- Original Message -----
From: "Tony Finch" <dot(_at_)dotat(_dot_)at>
To: "Hector Santos" <hsantos(_at_)santronics(_dot_)com>
Cc: <ietf-smtp(_at_)imc(_dot_)org>
Sent: Wednesday, June 29, 2005 3:18 PM
Subject: Re: Bounce/System Notification Address Verification
On Tue, 28 Jun 2005, Hector Santos wrote:
Ok, at 44 seconds, it became the CBV, as you can see thre wasn't a
response
to the RCPT TO: The timeout appears to be 35 seconds.
That's way too short for a CBV timeout, and in any case the appropriate
response to a timeout is 4xx (or perhaps 2xx), certainly not 5xx.
Tony,
When I first started our WCSAP system with the CBV feature in 2003, I
reseachd other similar systems in place. I found an old Dec/2003 file on my
drive that I saved clipped from Exim documentation regarding its Callout
feature. I have the whole Exim Callout section below. But here are the
highlights/questions.
| 37.15. Additional parameters for callouts,
|
| The default is 30 seconds. The timeout is used for each response from
| the remote host.
I don't recall if I decided to use 35 seconds based on Exim's 30 seconds.
But off hand, it reflects the default Windows socket timeout value.
| 37.14. Callout verification
|
| For a recipient address, the MAIL command contains the sender address of
| the incoming message. If the response to the RCPT command is a 2xx code,
| the verification succeeds. If it is 5xx, the verification fails. For any
| other condition, Exim tries the next host, if any.
We tried this next host idea for a timeout and found in cases where there
was a timeout, which was usually all the time, the spammers exhibited the
intentional noresponsive behavior. So imagine a spammer with 3-4 host,
and each will fail with the Exim 30 second timeout., we have a 90-120 delay
in the SMTP session creating scalability issues.
As I said, the false postive is the exception, not the rule, so currently a
timeout on the responses (not the connection, or a busy response at the
greeting) will reject the session.
I am going to reexplore this and maybe just add more options here:
CBVCommand.Timeout 45 // new default
CBVCommand.Timeout.ResponseCode 451 // explore new default
CBVCommand.Timeout.TryNextHost True // explore new default
| If there is a problem
| with all the remote hosts, the ACL yields ``defer'', unless the defer_ok
| parameter of the callout option is given, in which case the condition is
| forced to succeed.
Can you explain the key difference between "defer" and "defer_ok"?
What does "forced to succeed" mean?
Regarding the "Postmaster" and "random" test it offers, it says:
| The idea here is to try to determine whether the remote host accepts all
| local parts without checking. If it does, there is no point in doing
| callouts for specific local parts. If the ``random'' check succeeds, the
| result is saved in a cache record, and used to force the current and
| subsequent callout checks to succeed without a connection being made,
| until the cache record expires.
The problem with this is its not doing a random domain, just a random local
part. By far, due to spam problem awareness with Open Relays, only bad or
poorly setup systems keep these open. Even if its a "good guy," this poor
setup plays into the hands of bounce attacks. So it better to reject these
transactions. Of this is my technical opinion for protecting a system.
However, can you explain the postmaster usage?
As you know, this can cause a loopback. Who ever ask me why we have an
option to skip postmaster return paths, this is why. Although it only
causes 1 extra CBV, by RFC 2821 requirements to accept all RCPT
<postmaster> commands, a CBV to validate a postmaster is a waste of time.
So we skip it for backward compatibility reasons.
Comments?
--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com
--------------------------
Exim
37.14. Callout verification
For non-local addresses, routing verifies the domain, but is unable to
do any checking of the local part. There are situations where some means
of verifying the local part is desirable. One way this can be done is to
make an SMTP callback to the sending host (for a sender address) or a
callforward to a subsequent host (for a recipient address), to see if
the host accepts the address. We use the term callout to cover both
cases. This facility should be used with care, because it can add a lot
of resource usage to the cost of verifying an address. However, Exim
does cache the results of callouts, which helps to reduce the cost.
Details are in the next section.
A successful callout does not guarantee that a real delivery to the
address would succeed; on the other hand, a failing callout does
guarantee that it would fail.
If the callout option is present on a condition that verifies an
address, a second stage of verification occurs if the address is
successfully routed to one or more remote hosts. The usual case is
routing by a dnslookup or a manualroute router, where the router
specifies the hosts. However, if a router that does not set up hosts
routes to an smtp transport with a hosts setting, the transport's hosts
are used. If an smtp transport transport has hosts_override set, its
hosts are always used, whether or not the router supplies a host list.
When a host list is available, Exim makes SMTP connections to the remote
hosts, to test whether a bounce message could be delivered to a sender
address, or whether the current message could be delivered to a
recipient address. For a sender address, it sends:
HELO <primary host name>
MAIL FROM:<>
RCPT TO:<the address to be tested>
QUIT
For a recipient address, the MAIL command contains the sender address of
the incoming message. If the response to the RCPT command is a 2xx code,
the verification succeeds. If it is 5xx, the verification fails. For any
other condition, Exim tries the next host, if any. If there is a problem
with all the remote hosts, the ACL yields ``defer'', unless the defer_ok
parameter of the callout option is given, in which case the condition is
forced to succeed.
For SMTP callout connections, the port to connect to and the outgoing
interface are taken from the transport to which address was routed, if
it is a remote transport. Otherwise port 25 is used, and the interface
is not specified.
37.15. Additional parameters for callouts
The callout option can be followed by an equals sign and a number of
optional parameters, separated by commas. For example:
verify = recipient/callout=10s,defer_ok
The old syntax, which had callout_defer_ok and check_postmaster as
separate verify options, is retained for backwards compatibility, but is
now deprecated. The additional parameters for callout are as follows:
<a time>: This specifies the timeout that applies for the callout
attempt to each host. For example:
verify = sender/callout=5s
The default is 30 seconds. The timeout is used for each response from
the remote host.
defer_ok: Failure to contact any host, or any other kind of temporary
error is treated as success by the ACL. However, the cache is not
updated in this circumstance.
no_cache: The callout cache is neither read not updated.
postmaster: A successful callout check is followed by a similar check
for the local part postmaster at the same domain. If this address is
rejected, the callout fails. The result of the postmaster check is
recorded in a cache record; if it is a failure, this is used to fail
subsequent callouts for the domain without a connection being made,
until the cache record expires.
random: Before doing the normal callout check, Exim does a check for a
``random'' local part at the same domain. The local part is not really
random - it is defined by the expansion of the option
callout_random_local_part, which defaults to
$primary_host_name-$tod_epoch-testing
The idea here is to try to determine whether the remote host accepts all
local parts without checking. If it does, there is no point in doing
callouts for specific local parts. If the ``random'' check succeeds, the
result is saved in a cache record, and used to force the current and
subsequent callout checks to succeed without a connection being made,
until the cache record expires.
37.16. Callout caching
Exim caches the results of callouts in order to reduce the amount of
resources used, unless you specify the no_cache parameter with the
callout option. A hints database called ``callout'' is used for the
cache. Two different record types are used: one records the result of a
callout check for a specific address, and the other records information
that applies to the entire domain (for example, that it accepts the
local part postmaster).
When an original callout fails, a detailed SMTP error message is given
about the failure. However, for subsequent failures use the cache data,
this message is not available.
The expiry times for negative and positive address cache records are
independent, and can be set by the global options
callout_negative_expire (default 2h) and callout_positive_expire
(default 24h), respectively.
If a host gives a negative response to an SMTP connection, or rejects
any commands up to and including
MAIL FROM:<>
any callout attempt is bound to fail. Exim remembers such failures in a
domain cache record, which it uses to fail callouts for the domain
without making new connections, until the domain record times out. There
are two separate expiry times for domain cache records:
callout_domain_negative_expire (default 3h) and
callout_domain_positive_expire (default 7d).
Domain records expire when the negative expiry time is reached if callouts
cannot be made for the domain, or if the postmaster check failed. Otherwise,
they expire when the positive expiry time is reached. This ensures that, for
example, a host that stops accepting ``random'' local parts will eventually
be noticed.
The callout caching mechanism is based entirely on the domain of the address
that is being tested. If the domain routes to several hosts, it is assumed
that their behaviour will be the same.
37.17. Sender address verification reporting
When sender verification fails in an ACL, the details of the failure are
given as additional output lines before the 550 response to the relevant
SMTP command (RCPT or DATA). For example, if sender callout is in use, you
might see:
MAIL FROM:<xyz(_at_)abc(_dot_)example>
250 OK
RCPT TO:<pqr(_at_)def(_dot_)example>
550-Verification failed for <xyz(_at_)abc(_dot_)example>
550-Called: 192.168.34.43
550-Sent: RCPT TO:<xyz(_at_)abc(_dot_)example>
550-Response: 550 Unknown local part xyz in <xyz(_at_)abc(_dot_)example>
550 Sender verification failed
If more than one RCPT command fails in the same way, the details are given
only for the first of them. However, some administrators do not want to send
out this much information. You can suppress the details by adding
``/no_details'' to the ACL statement that requests sender verification. For
example:
verify = sender/no_details