spf-discuss
[Top] [All Lists]

Re: Re: [SPF v1 Draft] Last chance before I submit...

2004-10-13 05:53:44
In <416CDE41(_dot_)1490(_at_)xyzzy(_dot_)claranet(_dot_)de> Frank Ellermann 
<nobody(_at_)xyzzy(_dot_)claranet(_dot_)de> writes:

wayne wrote:

in the record "v=spf1 a k$", if the "a" mechanism passes,
the invalid mechanism "k$" is never detected.

k$ _is_ a syntax error, because there's no $ in a "name",
and therefore implementations MAY throw a PermError (4.6).

There is no "MAY" in section 4.6.  There is no reason that there
shouldn't be a "MUST" saying, requiring all syntax errors to be
detected.


2)  Unknown mechanisms are a bad idea and are never used

Unrecognized mechanisms throw a PermError (7.1), that's an
excellent reason to never use any unrecognized mechanisms.

Agreed.

The language in the draft is very careful about this issue,
a future version will most probably use a new tag like spf2.0
or v=spf2 for sender policies with new mechanisms.

Yes..

If you have implemented a solution, where "foo" is handled
like "k$", then you're technically violating 7.1.  But in
practice nobody plans to introduce new mechanisms in v=spf1,
so that's only a theoretical problem.

It is not at all theoretical because of all the typos and bogus stuff
in SPF records.   Yes, in theory, this is a theoretical problem, in
practice, it is a practical problem.  The practical result of allowing
unknown modifiers is a less reliable SPF system.


Unknown mechanisms are not an effective method of extending
the SPFv1 semantics.

Yes, that's an obvious consequence of 7.1.

things like "prt:" instead of "ptr:" or "ipv4" instead of
"ip4".

Come on, you know the reason for 7.1, it's not about adding
nonsense in later versions with the old v=spf1 tag, it's about
left-to-right evaluations.  If you say that your code won't
accept mechanisms "foo", "prt", or "ipv4", then what you
really want is a SHOULD instead of the MUST in 7.1.  I'd be
happy with this solution.

Excuse me?

Have you read what Meng and Mark have written about unknown mechanism
on this list?   Have you read section 7.1?

Unknown mechanisms *IS* all about trying to be able to extend stuff
with the v=spf1 tag.  It is a bad idea.  Unknown mechanisms can't do
anything that unknown modifiers can't do better.

There is only one "MUST" in section 7.1, and that deals with
modifiers, not mechanisms and I think it should stay asis.



an SPF implementation that is safe from being used for DDoS
attacks *MUST* violate the SPF standard because it can not
implement and evaluation the check_host() function in
accordance with section 4.

No idea what you're after here, please elaborate. 

See, for example:
http://archives.listbox.com/spf-discuss(_at_)v2(_dot_)listbox(_dot_)com/200312/0393.html
http://archives.listbox.com/spf-discuss(_at_)v2(_dot_)listbox(_dot_)com/200404/0286.html
http://archives.listbox.com/spf-discuss(_at_)v2(_dot_)listbox(_dot_)com/200405/0083.html
http://www.imc.org/ietf-mxcomp/mail-archive/msg01263.html
http://www.imc.org/ietf-mxcomp/mail-archive/msg01944.html

I know I've discussed these problems more times than the above, but
I got bored searching and these should do as an example.

Yes, all of these were posted before you became involved with MARID,
let alone SPF, but they do not predate Meng and Mark's involvement.  I
do not see any reason to keep reposting stuff like this every time
someone new comes along.


To quote from one of the above messages:

Basically, what implementers need to do is look through their code and
try to prove that the execution time, memory requirements and I/O
requests are always bounded. Then, look through their code and see if
a cracker can use the SPF checks to amplify a DoS attack. 


                                                   Or better
propose something for the "security considerations" section.

Been there, done that, was ignored.  But, just for grins, I'm posting
a patch that I sent to Meng for draft-spf-200405.txt


I'm not too thrilled with mentioning well-known attacks if
you can propose something more specific for SPF.

As mentioned in some of the above posts, libspf2 (aka libspf-alt) has
been imune to these attacks for a long time.



dos.midwestcs.com,

dos.midwestcs.com       text = "v=spf1 mx:a.%{d} mx:b.%{d}
 mx:c.%{d} mx:d.%{d} mx:e.%{d} mx:f.%{d} mx:g.%{d} mx:h.%{d}
 mx:i.%{d} mx:j.%{d} -all"

Resulting in 10 * 13 MX (?)  Tiny little mazes, all similar.
None of these 130 MXs exist, is that the idea ?

Right.  Remember that NXDOMAIN has a much shorter TTL than the SPF
and MX records.  So, once SPF becomes common, you simply take out your
list of domains, send SMTP commands to each domain in turn.  You can
start over after 5 minutes (NTTL for most name servers), and all the
attacker's DNS lookups will be cached, but the victim will be blasted
again.

Also note that the above is just an example and doesn't come anywhere
near the amplification factor that is allowed by the SPF spec.

Yes, I have measured the actual TCP/UDP traffic and calculated the DoS
amplification factor.  No, I don't still have those numbers.  Feel
free to do your own work.


domain owners must publish SPF records for ever sub-domain.
Very few domain owners realize this.

IMHO covered by the wildcard discussion, what else could you
say about it ?

No, it is *not* well covered by the wildcard discussion in the spec.
Read spf-draft-200406.txt


The %{h} macro-variable has been removed.

Sure, check_host() doesn't know what it's doing.  And "we"
(tinw) wanted to be free to define HELO stuff later in any
appropriate way.  There was no consensus to keep it "as is",
and it was only optional.

There are people out there with %{h} macros.  It was primarily MS that
wanted it removed.  


There is *no* good reason to remove it from the SPFv1 spec
and create an incompatible change.

IBTD.  One evaluation of check_host() is good enough for the
purpose of SPF1.  You've already mentioned DDoS scenarios,
adding another check_host() for the HELO doesn't help.  And
the old optional HELO check wasn't a complete solution for
forwarders, or for the HELO sanity checks proposed by Hector.

You are confusing the removal of the %{h} macro with the removal of
the HELO checking.  But, several of us strong disagree that HELO
checking that was allowed in previous drafts was not useful.


The HELO stuff is too important to "burn" it in a SPF1 memo.
SPF1 is about MAIL FROM.  Not about PRA / HELO / SUBMIT / foo.
Let's discuss HELO for SPF2.

Your idea of "SPF1" seems to be based off of much more recent stuff.
HELO for SPFv1 was discussed long before you became involved and was
reasonably well settled.  Then, it was removed when Mark became
involved again with the editing the MARID drafts.



As mentioned above, here is a patch I sent Meng when Meng and I were
working on spf-draft-200404 to deal with the DoS situation: (Mark was
not involved during that time period)


--- spf-draft-200404.1.txt      2004-04-25 18:31:40.000000000 -0500
+++ spf-draft-200404.1.wrs.txt  2004-04-25 22:50:32.000000000 -0500
@@ -166,6 +166,19 @@
    Hence, a record with a version of "v=spf10" is not considered a
    record with version "v=spf1".
 
+   The length of the SPF record SHOULD be short enough so that the DNS
+   query can be returned in a single UDP packet.  Records that are too
+   long MAY be silently ignored.
+
+   This effectively limits the length of the SPF record to somewhere
+   around 400-450 characters.  SPF records that do not fit into the
+   500 character UDP DNS packet must be transmitted via TCP.  Some
+   older name servers do not support TCP, but a more common problem
+   is that often firewalls do not allow DNS over TCP since it is
+   rarely used.  So, a mail admin may not even be aware that longer
+   records are being ignored.
+
+
    An example SPF record is:
    
       v=spf1 +mx +a:colo.example.com/28 -all
@@ -593,8 +606,9 @@
     
    SPF clients first perform an MX lookup on the <target-name>.  SPF
    clients then perform an A lookup on each MX name returned, in order
-   of MX priority.  The <sending-host> is compared to each returned IP
-   address.  If any address matches, the mechanism matches.
+   of MX priority.  A limit of 5 MX names MUST be enforced.  The
+   <sending-host> is compared to each returned IP address.  If any
+   address matches, the mechanism matches.
 
    Note Regarding Implicit MXes: If the <target-name> has no MX records,
    SPF clients MUST NOT pretend the target is its single MX, and MUST
@@ -612,11 +626,13 @@
 
    First the <sending-host>'s name is looked up using this procedure:
    perform a PTR lookup against the <sending-host>'s IP.  For each
-   record returned, validate the host name by looking up its IP address.
-   If the <sending-host>'s IP is among the returned IP addresses, then
-   that host name is validated.  In pseudocode:
+   record returned, validate the host name by looking up its IP
+   address.  If the <sending-host>'s IP is among the returned IP
+   addresses, then that host name is validated.  A limit to checking 5
+   PTR records MUST be enforced.  In pseudocode:
 
      sending-host_names := ptr_lookup(sending-host_IP);
+     if more than 5 sending-host_names are found, use at most 5.
      for each name in (sending-host_names) {
        IP_addresses := a_lookup(name);
        if the sending-host_IP is one of the IP_addresses {
@@ -782,10 +798,17 @@
    
        explanation = "exp" "=" domain-spec
 
-   When an SPF client performs a query, and the result is anything other
-   than pass, then the explanation string, if present, SHOULD be
+   When an SPF client performs a query, and the result is anything
+   other than pass, then the explanation string, if present, SHOULD be
    presented to the SMTP client after macro expansion.  See section 7.
    
+   The SPF client MAY limit the length of the explanation.  For
+   example the MTA may have a maximum length of that an error message
+   can be.  The SPF client SHOULD make it clear when an explanation
+   string is coming from a third party.  For example, it can add the
+   macro string "%{d} explains: " to the beginning of the explanation
+   string. 
+   
    Suppose example.com has this SPF record
 
       v=spf1 mx -all exp=explain._spf.%{d}
@@ -890,21 +913,47 @@
 
 6.2 Processing Limits
    
-   During processing, an SPF client may perform additional SPF
-   subqueries due to the Include mechanism and the Redirect modifier.
+   During process, and SPF client may perform DNS lookups and
+   additional SPF subqueries due to the "include" mechanism and the
+   "redirect" modifier.
+
+   In order to prevent Denial-of-Service attacks, the total DNS
+   lookups must be limited.  The subject of a Denial-of-Service attack
+   can be either the SPF client directly, or some third party domain
+   that is referenced in the SPF record.  For example, a malicious
+   person could create an SPF record with many references to a victim
+   domain, send many emails to different SPF clients and the SPF
+   clients would create a DoS attack.  In effect, the SPF clients are
+   being used to amplify the attacker's bandwidth by using few bytes
+   to send the email than is generated by the DNS queries.
+
+   As a result, limits that may seem reasonable for an individual mail
+   server can still allow an unreasonable amount of bandwidth
+   amplification.  Therefore the processing limits need to be quite
+   small. 
+
+   SPF implementations MUST limit the number of mechanisms that do DNS
+   lookups to at most 10.  For example, the "mx" mechanism requires a
+   DNS lookup, so will count against this limit, while the "ip4"
+   mechanism does not require any DNS lookups.
+
+   When evaluating the "mx" mechanism, there MUST be a limit of no
+   more than 5 mail looked up and checked for matching IP addresses.
+
+   When evaluating the "ptr" mechanism or the %{p} macro variable,
+   there MUST be a limit of at most 5 PTR DNS records looked up and
+   checked for a validated hostname.
+
+   SPF implementation SHOULD limit the total amount of data obtained
+   form the DNS queries.  For example, when DNS over TCP is available,
+   there may need to be an explicit limit to how much data will be
+   accepted to prevent excessive bandwidth usage or memory usage.
+
+   SPF implementation SHOULD impose a walk-clock time limit on the
+   evaluation of an SPF record in order to prevent problems with DNS
+   queries from creating excessive delays.  The limit SHOULD be allow
+   the SPF evaluation to last at least 30 seconds.
 
-   SPF clients must be prepared to handle records that are set up
-   incorrectly or maliciously.  SPF clients MUST perform loop detection,
-   limit SPF recursion, or both.  If an SPF client chooses to limit
-   recursion depth, then at least a total of 20 redirects and includes
-   SHOULD be supported.  (This number should be enough for even the most
-   complicated configurations.)
-   
-   If a loop is detected, or if more than 20 subqueries are triggered,
-   an SPF client MAY abort the lookup and return the result "unknown".
-
-   Regular non-recursive lookups due to mechanisms like "a" and "mx" or
-   due to modifiers like "exp" do not count toward this total.
    
 6.3 The Received-SPF header
 
@@ -1002,7 +1051,7 @@
    Certain directives perform macro interpolation on their arguments.
 
      macro-string = *( macro-char / VCHAR )
-     macro-char   = ( "%{" ALPHA *DIGIT [ "r" ] *delim "}" )
+     macro-char   = ( "%{" ALPHA *DIGIT [ "r" ] *delimiter "}" )
                     / "%%"
                     / "%_"
                     / "%-"
@@ -1017,54 +1066,85 @@
       s = responsible-sender
       o = responsible-domain
       d = current-domain
-      i = SMTP client IP
+      i = SMTP client IP (nibble format when an IPv6 address)
+      c = SMTP client IP (easily readable format)
       t = current timestamp in UTC epoch seconds notation
       p = SMTP client domain name
       v = client IP version string: "in-addr" for ipv4 or "ip6" for ipv6
       h = HELO/EHLO domain
+      r = receiving domain
 
    The uppercase versions of those macros are URL-encoded as well.
 
    A '%' character not followed by a '{', '%', '-', or '_' character
-   MUST be interpreted as a literal.  SPF publishers SHOULD NOT rely on
-   this feature; they MUST escape % literals.  For example, an
-   explanation TXT record
+   MUST be interpreted as a literal, with the exceptions of when
+   follow character is either a " " (space), or the end of the record.
+   In those cases, the '%' must be interpreted as if it was "%%".  SPF
+   publishers SHOULD NOT rely on this feature; they MUST escape '%'
+   literals.  For example, an explanation TXT record
       Your spam volume has increased by 581%
    is incorrect.  Instead, say
       Your spam volume has increased by 581%%
 
-   Legal modifiers are
+   Legal optional modifiers are:
 
-        *DIGIT ; one or more digits
+        *DIGIT     ; zero or more digits
         'r'    ; reverse value, splitting on dots by default
+        delimiters ; zero or more delimiter characters
 
-   The DIGIT modifier indicates the number of right-hand parts to use
-   after optional reversal.  The modifier MUST be nonzero.  If DIGIT
-   specifies more parts than are available, all the available parts are
-   used.  If the DIGIT was 5, and only 3 parts were available, the macro
-   interpreter would pretend the DIGIT was 3.
+   The macro strings are always split into parts.  After performing
+   any reversal operation or chopping off the left-hand parts, the
+   parts are always rejoined using "." and not the original splitting
+   characters.
+   
+   By default, strings are split only on "." (dots), which, when
+   joined back together, will result in no changes to the string.
+   Modifiers may be followed by one or more delimiter characters which
+   are used instead of the ".".  Splitting characters MUST be one or
+   more of the characters: ".", "-", "+", or "=".
 
    The 'r' modifier indicates a reversal operation: if the client IP
    address were 192.0.2.1, the macro %{i} would expand to "192.0.2.1"
    and the macro %{ir} would expand to "1.2.0.192".
 
-   The DIGIT and the 'r' modifiers split a string into parts.  By
-   default, strings are split on "." dots.  Modifiers may be followed by
-   one or more splitting characters which are used instead of the ".".
-   Splitting characters MUST be non-alphanumeric.  Parts are always
-   rejoined using "." and not the original splitting characters.
+   The DIGIT modifier indicates the number of right-hand parts to use
+   after optional reversal.  If the modifier is specified, it MUST be
+   nonzero.  If no DIGITs are specified, or if the value specifies
+   more parts than are available, all the available parts are used.
+   If the DIGIT was 5, and only 3 parts were available, the macro
+   interpreter would pretend the DIGIT was 3.  Implementations MAY
+   limit the number, but MUST support at least a value of 9.
 
    For the "l" and "s" macros: when the local-part is not defined, the
    string "postmaster" is substituted.  The local-part might be
    undefined if the <current-domain> is drawn from the HELO command
    rather than the MAIL FROM.
 
+   For IPv4 addresses, both the "i" and "c" macros expand to the
+   standard dotted-quad format.  For IPv6 addresses, the "i" macro
+   expands to an IP address in a format such that "%{ir}.%{v}.arpa"
+   forms a suitable reverse DNS domain name.  The "c" macro expands to
+   an IP address as specified in RFC3513, suitable for easily read.
+
+   The "t" macro MUST not be allowed in mechanisms as this would
+   greatly reduce the effectiveness of DNS caching.  The "t" macro
+   MUST be allowed in modifiers and in the explanation records.  The
+   value of the "t" macro SHOULD remain the same during the evaluation
+   of an SPF record.
+
    The "p" macro expands to the validated domain name of the SMTP
    client.  The validation procedure is described in section 4.6.  If
    there are no validated domain names, the word "unknown" is
    substituted.  If multiple validated domain names exist, the first one
    returned in the PTR result is chosen.
 
+   The "r" macro expands to the domain name of the receiving MTA.
+   This SHOULD be a fully qualified domain name, but if one does not
+   exists (as when the checking is done by a script), the word
+   "unknown" is substituted.  The domain name MAY be different than
+   name found in the MX record that the client MTA used to locate the
+   receiving MTA.
+
    The "s" macro expands to the sender email address: a localpart, an @
    sign, and a domain.  The "o" macro is the domain part of the "s".
    They remain the same during a recursive "include" or "redirect"
@@ -1404,9 +1484,19 @@
    introduced to ESMTP to require a simple challenge/response
    confirmation.
 
-   SPF clients need to limit the number of includes and redirects to
-   avoid attacks.  Implementations are required to follow recursion to a
-   minimum of 20.
+   SPF clients need to limit the number of DNS queries in order to
+   prevent Denial-of-Service attacks to both the SPF client and to
+   third parties that are referenced in the SPF record.  See section
+   6.2 "Processing Limits" for details.
+
+   SPF uses information supplied third parties, such as the HELO
+   domain name, the envelope-from address and SPF records.  This
+   information is then sent to the receiver in the Received-SPF: mail
+   headers and possibly returned to the client MTA in the form of an
+   SMTP rejection message.  This information must be check for invalid
+   characters and excessively long lines.  It must be made clear which
+   information is coming from third parties.
+
    
 
 11. IANA Considerations
@@ -1538,11 +1628,11 @@
     ip6-cidr-length  = "/" 1*DIGIT
 
     macro-string = *( macro-char / VCHAR )
-    macro-char   = ( "%{" alpha *digit [ "r" ] *delim "}" )
+    macro-char   = ( "%{" alpha *digit [ "r" ] *delimiter "}" )
                    / "%%" / "%_" / "%-"
 
     name        = alpha *( alpha / digit / "-" / "_" / "." )
-    delim       = "." / "-" / "+" / "," / "/" / "_" / "="
+    delimiter   = "." / "-" / "+" / "="
     
 Appendix B. Extended Examples