Re: Differences between CSV and Sender-ID
2004-07-02 03:35:36
Hi Dave,
I'm not exactly pleased to see my post carved up, with a number of 1-3 line
responses added for every 1-3 lines I wrote. I feel it is a style that
encourages fighting, bickering, and disagreement rather than mutual
understanding and consensus-building. However, I don't believe at all that
you meant to be rude or anything. Perhaps you interpreted my message as an
attack against CSV and reacted defensively?
Anyway, you raise some good points here, so I will attempt to reply as
concisely as I can. It's going to appear disjointed but I will try not to
break things up too much further than they already are.
--Dave Crocker <dhc(_at_)dcrocker(_dot_)net> wrote:
Greg,
GC> A HELO check can catch some really obvious bad cases (like spam or
viruses GC> using the receiver's own name) and some obvious good cases
(like people we GC> want to whitelist).
With respect to CSV, I think this needs to be phrased quite
differently, even though the semantics will probably seem similar, or
at least not contradictory. The reason for this need is that I think it
leads to a very different perspective on the implications of a CSV
mechanism versus an SPF-like mechanism.
HELO makes an assertion about the operation and accountability of the
MTA. There is quite a bit of history and current use of services that
vet sending SMTP clients and their network operators.
A HELO mechanism check can be used to produce a domain-name based
codification of such checking, rather than requiring that white/black
lists be maintained in terms of IP Addresses. The benefit of having a
domain name base involves all of the reasons we all like domain names
better than IP Addresses, for use by humans.
Without commenting on mechanisms, I totally agree with your explanation of
HELO and its significance. I was attempting to keep it a bit simple for
the readers. The two paragraphs above are a good explanation as to why
HELO is significant, and why checking it (by whatever mechanism) is
desirable. All is well.
GC> CSV also uses HELO to tie a reputation to the sending MTA.
The concern about accreditation (of which "reputation" is a subset) is
rather interesting, here. What folks seem to be missing is that ALL
mechanisms that involve acceptance or rejection based on a name or
address has an accreditation component. Accreditation is, in fact, the
acceptance or rejection policy engine.
CSV merely specifies two external standards for such a mechanism.
So, yes, a mechanism that only seeks to detect forgeries does not have
an accreditation mechanism. But forgive me if I am wrong: I thought
folks were interested in detecting and preventing spam, and spam is
very, very much about accreditation, not forgery.
Forgery is a current symptom, rather than a core aspect of spam and
virus sending. Eliminate forgery and there will still be masses of
spam and virus sending.
I had rather hoped we were trying to get at core issues nof fighting
spam, what with the scale of the problem and the cost and delay
inherent in any standards effort.
My statement wasn't meant as a negative. I actually agree with what you
said here. CSV hangs reputation/accreditation on HELO, SPF chooses to hang
it on another identity. I certainly didn't mean to imply that
accreditation is not important.
GC> This seems to
GC> be based on the assumption that good mail comes from good MTAs, and
bad GC> mail comes from bad MTAs, which some have suggested is not
well-supported.
"Some have suggested" is language that applies to any interesting
topic, for every possible point of view on the topic. Hence, it does
not carry any useful information. (Yeah, I am saying that a bit
sharply. It is a pet-peeve of mine about so-called news reporting and
I really hope the content-free utterance does not seep into serious
technical discussions.)
To move this particular point into something that might be productive,
please refer to the thread "who are we accrediting?" and note John
Levine's posting. I'll post a response to it.
In the part you quoted, I was trying to point out one area of disagreement,
without actually taking sides (I explained my own opinion after the "My
opinion" tag), so my apologies for the vagueness.
So, to be clear, I am suggesting that this assumption is not valid or
useful. I think the reputation of the MTA is often interesting, but
certainly not enough in itself to judge the quality of the mail. But,
after reading your message I am starting to think that you don't believe
this assumption either, that there are only "good" and "bad" MTAs. In that
case, this disagreement is not directed at you, but at other CSV supporters
(Matthew and Doug) who have been suggesting that checking HELO against a
reputation is pretty much all you need and other proposals that check other
identities are worthless, doomed to failure, or both.
GC> I think checking the HELO *alone* is not an adequate solution to the
GC> problem set.
I agree. RFC2822 author/sender based accreditation is also going to
be needed. The nature and form of that accreditation is a different
question.
Right... that's what I was trying to say too :)
GC> If the main thing we want out of MARID is to stop people forging mail
I do not have access to the working group charter as I write this, but
I sure hope that forgery is not the primary concern of the working
group.
Otherwise, there is a rather large community of email users and
providers who are going to rather upset that we spent all this time
and did nothing that is intended to reduce spam.
On the other hand, I could see how "DNS-based MTA authentication" could
cause one to think that forgery is the focus.
Wow, we're interrupting mid-sentence now, I see :) I won't spend much time
on this one other than to say:
1. I want to stop spam too, and I think stopping forgery is a necessary
but sufficient step..
2. I honestly believed that stopping forgery was the point of the WG and
that stuff that stops spam by other means than stopping forgery would be
ruled out of scope, and
3. It looks like you agreed with the important part of my sentence anyway
:)
GC> apparently-from and bouncing-to our own domains, a MAILFROM/PRA check
is GC> going to be required.
Some sort of rfc2822 author/sender accredition is going to be
required... in some cases.
Agreed :)
GC> Mechanically, CSV and SPF are both capable of checking HELO.
Mechanically, CSV and SPF are both fruit. But let me tell you, you do
not want to think about or use durian the same way you think about and
use oranges.
However, your statement highlights a deeper problem in most of the
efforts to discuss CSV and SPF differences: Such efforts are almost
entirely tied to mechanical and syntactic issues and do not focus on
underlying concepts.
Right... That actuall IS what I mean here -- I mean to separate the
mechanics of each proposal from the underlying concepts. The assertion I
was trying to test is whether the mechanism SPF uses to test PRA, MAIL FROM
and HELO is capable of doing the same things the CSV mechanism does.
If it cannot for strictly *mechanical* reasons, I would like to understand
what they are. So far the answer whenever I ask this is "Well, you COULD
use SPF TXT records instead of SRV records, but why would you want to?" If
it's possible to present an end user with one tool that has two
applications, that might be a worthwhile goal. Speaking only of the
*mechanism* I don't see that SRV records have an inherent advantage over
TXT records, or that the underlying concepts of CSV depend on SRV records.
CSV and SPF are fundamentally different pardigms.
CSV vets an MTA's traffic.
SPF vets an RFC2822 author/sender's message.
They are orthogonal informational-theoretic areas of consideration.
Where the confusion comes in, of course, is that SPF involves the MTA,
albeit through an indirection.
Let's try for some concise descriptions of the two paradigms:
SPF:
Per-message MTA path validation, based on Author/Sender
authorization and accreditations.
CSV:
MTA traffic validation, based on MTA operator authorization and
accreditation.
SPF vets an MTA's sending a single message. It accredits the MTA
based on the RFC2822 author/sender. While introducing a
path-dependency into the mechanism, it simply defers the hard
question, namely accrediting the author/sender.
CSV vets an entire MTA session. It accredits the MTA based on the
operator of that MTA.
I don't really agree that CSV and SPF are fundamentally different
paradigms. They are different, but I don't think fundamentally so, and I
don't think either of them represents a "paradigm" really.
I think of SPF not as a great idea, but as a collection of great ideas.
Some of these are:
- A mechanism that maps (domain name, IP) onto (pass, fail, unknown)
- Application of this mechanism to MAIL FROM, to vet a message path (or
partial path, when forwarders use SRS)
- Application of this mechanism to PRA, to vet a message path (or partial
path, when forwarders use recommended headers)
- Accreditation can be applied to the domain of any ID that returns pass
result.
- An ID that returns fail result should be treated as highly suspect and
probably rejects. An ID that returns unknown result should not be used to
judge a mail as good or bad and the receiver should fall back to other
methods.
CSV is also made of multiple great ideas, such as:
- A mechanism that maps (domain name, IP) onto (allowed, disallowed,
no_info)
- Application of this mechanism to HELO, to vet an MTA
- Accreditation can be applied to the MTA based on its name, if result is
allowed.
- An IP address specifically disallowed from using the name claimed in
HELO should be treated as MTA-not-grata
- An MTA that has no info CSV may check should be rated on other means
(e.g. IP) or not at all.
The point of this exercise is to separate the "mechanism" carrying the
message from the content and meaning of the message itself. The SRV record
mechanism is clever, but I got the feeling from reading CSV documents and
speaking to you and other CSV supporters that it is not the main important
thing that CSV does.
By suggesting that the mechanisms *could be* compatible, I don't mean to
imply that the two types of checks already mean the same thing. They
don't. SPF has a couple of modes where it checks HELO, but it lacks an
explanation as to why one might want to do that, what the information
means, and how to interpret it and act on it.
Why am I so keen to show that one mechanism could be used for both checks?
Well, one of the first things that this WG worked on was deciding which
identities to check. My understand was, at the time, that there was a
pretty strong consensus that we should work on both 2821 and 2822
identities, and I *thought* we had also decided that if we tackle one
identity first, we would do so in such a way that the other identities
could be checked with the same or similar mechanism.
Current whitelist and blacklist services focus on the MTA network, ie,
the operator of the MTA. So CSV provides a standardizing mechanism
for existing practise.
The limitations of that practise are demonstrated every day, but so
are the benefits.
That is an excellent point, and I agree.
GC> - If the MTA name is also used as a HELO name for one of the MTAs
GC> - In most cases the existing SPF record should be sufficient,
since GC> it probably includes that MTA.
My guess is that you are talking about the narrow case in which the
RFC2822 author/sender has the same domain name as the MTA HELO. While
a popular scenario, it is a long way from being the ONLY popular
scenario. And that's the problem. SPF is problematic for a number of
other such popular scenarios.
You are correct, that should have been "sender domain name" not MTA name.
I agree, this is definitely not the majority case. I mention it here
because it is really the only case where the SAME name may be used by both
email addresses and HELO. If the same name might be used by an MTA and by
the RHS of an email address, I think chances are very good that the allowed
IPs for both cases will be the same.
Again, this hearkens back to the discussion of which identities we want to
be able to validate. At the time, we identified HELO, MAIL FROM and
From:/Sender:, and along with the idea that perhaps all three merit
checking, we brought up the cases where the same domain name might be used
in different contexts. Each context might have wildly different meaning
and usage, but where the NAME is exactly the same, the set of authorized
IPs would usually be the same or a blend of the two usage sets would be
suitable. If I remember correctly, not everyone was convinced at the time
that a single set of IPs would always work, so there was some discussion of
a "scope tag" of sorts, but I think most of the group agreed at the time
that the need for this would be rare.
GC> Semantically, there is some difference in the understanding between
what GC> the CSV check means, and what the SPF+HELO check means.
It is rather more than "some".
Agreed. But if the implication is that they are different enough to
*require* different mechanisms, I would not agree with that.
Let me say this again because I think it is important:
THE IDEA OF USING ONE MECHANISM TO VALIDATE DIFFERENT IDENTITIES IS NOT NEW.
As I continue to suggest that CSV *could* be implemented using SPF TXT
records, people continue to look at me as if I'm speaking heresy. All I
can say is, please review the archives. This same WG agreed that multiple
identities are worthy of checking, and if possible they should be checked
in the same or similar ways. Did I misunderstand, or have we changed
directions on this, or has everyone just forgotten what we talked about for
the first month or more?
GC> It would be better to use ?include:comcast.net or ?ptr:comcast.net.
That GC> way the mail from those domains is still allowed, but not
"guaranteed" to GC> be from you.
This begs for an obvious question: What is the benefit to the
anti-spam world of something which offers no guarantees? Is that not
the same as saying "I enforce no anti-spam policies, since anyone can
claim to be part of my domain"? No accountability is a rather serious
deficiency.
To quote Douglas Adams, "We demand rigidly defined areas of doubt and
uncertainty!" :)
Seriously though, the "unknown" state was put in there for a reason. In an
ideal world, all my users would phone home and submit with SMTP AUTH and
all our mail would go out the pre-defined block of IPs. But, some domain
owners might want partial coverage, and might need some usage cases to be
supported in "legacy mode" for a while. If a domain owner is not 100% sure
he has rounded up all the roaming users, he may choose to write ?all at the
end - in which case forgeries would not be stopped, but the +entries in the
list can still be used to invoke reputation and whitelisting. If all the
roaming users happen to be on comcast.net, a record with ?ptr:comcast.net
-all is much better than ?all -- possibly enough to make spammers/forgers
move on to the next target.
In other words, the "unknown" state is a feature, not a bug. If you don't
agree, fine, don't use the feature. Your characterization of this mode as
a "deficiency" is uncharitable and seems to contain a high FUD to fact
ratio.
If you had not taken the sentence out of context, my original intent would
be a bit clearer -- I was actually responding to some other FUD based on a
wrong understanding of SPF (or intentional misreading or other straw man).
The example given by Doug and Matt both was "Well what about a domain that
publishes include:comcast.net? That means anyone on comcast.net could HELO
as my own name!" Yes, and this would be a mistake on the part of the
domain owner; they are in effect saying "We trust comcast.net to not forge
mail from us or otherwise use our name improperly."
GC> If the result comes back unknown, you can't attach reputation
GC> or whitelisting to that transaction, you just have to proceed in
"legacy" GC> mode.
And the value-add of SPF, in this scenario is what, exactly?
What does the administrator of the domain and/or the operator of the
receiving SMTP client get for their effort?
See above regarding FUD.
I will note also that despite disparagement pointed at the "unknown" mode
of SPF, CSV also has a de-facto "unknown" mode - you can just choose not to
publish any records at all for that particular name. I would assume that
reputation would not attach in this case either.
Is it clear to you that CSV has definite security advantages
over SPF/Sender-ID?
GC> There is general agreement that the smaller problem
GC> of HELO checking
"smaller problem"?
I hope you do not mean that identifying spam spigots is a small
problem or that doing it will be a small benefit.
That is one of the things CSV is useful for, that SPF is not. Entire
networks of compromised machines can be blocked with a single
accreditation entry, no matter what the domain names they use for their
rfc2822 author/sender.
Actually I was not referring to my own opinion as "general agreement" -- I
was referring to the decisions of this WG as to which identities should be
checked. I believe it was agreed that 2821.MAIL FROM was most important,
followed by 2822.From/Sender, and 2821.HELO was the least important of the
three.
I DO agree though; identifying spam spigots is a big problem, and doing it
is a big benefit. You have made a good case for HELO checking. I'm not
suggesting that HELO checking shouldn't be done, and you're not suggesting
that it's a total solution in itself that trumps others. We may not be on
the same page but I think it's in the same book :)
GC> CSV may have a better security story, but I believe this is a direct
result GC> of deciding to include fewer features and less flexibility.
Methinks there is a lesson in protocol design, here.
GC> Regarding DDOS concerns, I think they can be solved by placing some
limits GC> on the amount of recursion possible and the total number of
queries needed GC> per mail message, and that should satisfy most
concerns.
Offhand, I am not sure what you mean and I am certain I do not
understand how it pertains to protection against DDOS attacks.
Sorry, I didn't mean "[all] DDOS attacks" -- this was a specific reply to a
specific concern in SPF.
I don't know why Matt and Doug chose to say over and over again how nasty
and yucky and vulnerable SPF is, citing this as a reason why CSV is cool
and wanted and necessary. I don't think SPF and CSV are mutually exclusive
and I don't think the "air of competition" serves any of us well. I think
CSV contains great ideas and so does SenderID.
GC> I think there is enough consensus in the group that we need to
protect PRA GC> and/or MAIL FROM,
There is agreement that we need a mechanism that identifies and
accredits rfc2822 author/sender IN SOME CASES.
No comment at this time, Senator. :)
GC> and that HELO is of secondary importance.
I'm not sure whether you noticed, but there is a rather different tone
in the comments about HELO checking now than there was a month or so
ago.
I noticed. CSV has done a lot to bring HELO into the spotlight. This is a
good thing. As long as CSV isn't trying to elbow other proposals out of
the way, I don't have a problem with it.
In fact, Unified SPF is based pretty strongly on my efforts to get HELO
placed more prominently on SPF's radar screen. I have gone from not taking
HELO seriously to actively preaching the gospel of HELO to spf-discuss and
#spf on IRC.
I don't think HELO has eclipsed PRA/MAIL FROM/SUBMITTER in importance or
utility.
GC> Not everyone
GC> agrees with this, but I think a majority of folks think that HELO
checking
I am confused. Were the chairs asking each of us to perform a rough
consensus assessment of the working group?
I was referring again to the rough consensus reached at our first-phase
milestone. I believe it was agreed that 2821.MAIL FROM was most important,
followed by 2822.From/Sender, and 2821.HELO was the least important of the
three. Perhaps I misremember, but that's what I thought we said.
GC> So, if we are going forward with PRA/SenderID or
GC> something like it, it should be easy enough to adapt it to HELO
checking as GC> well.
I'm sure we all look forward to the specification that satisfies your
expectation.
Being worked on. I think you will be pleased. As I said before, if
Unified SPF borrows some ideas from CSV, please consider that a form of
flattery :)
--
Greg Connor <gconnor(_at_)nekodojo(_dot_)org>
<Prev in Thread] |
Current Thread |
[Next in Thread>
|
- Re: Differences between CSV and Sender-ID, Dave Crocker
- Re: Differences between CSV and Sender-ID, Dave Crocker
- Re: Differences between CSV and Sender-ID, Hector Santos
- Re: Differences between CSV and Sender-ID,
Greg Connor <=
- Message not available
- Re: Differences between CSV and Sender-ID, Dave Crocker
- Forging (was Re: Differences between CSV and Sender-ID ), Alan DeKok
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Dave Crocker
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Alan DeKok
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Eric A. Hall
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Dave Crocker
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Eric A. Hall
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Dave Crocker
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Eric A. Hall
- Re: Forging (was Re: Differences between CSV and Sender-ID ), Dave Crocker
|
|
|