I have to strongly disagree with the tone of this review of the review
document.
(Which is not to say that Ted isn't entitled to provide the review, or
that he is not entitled to a reasonable response.)
There are several issues that get mixed up together in defining a late
external review process. By definition, this is an external review. So
it is not reasonable to say that the reviewer should think like a WG
member, or have the full email conversations available.
So, the first question is what should happen when a reviewer says "X
does not make sense" or "Y seems to be assumed without explanation."
Some of the time, that is because that context is provided by other
documents which are reasonable to have read. Someone needs to say that.
(If a reviewer has become confused, likely some AD sill also become
confused. Maybe the sponsoring AD will realize that the solution is
some other document that is already referenced. But then again, maybe
he won't.)
An awful lot of the time though, the issue is one of "common knowledge".
Sometimes bluntly stated as "all the participants understand that Z,
which means ..." Well, the fact that working group members know Z does
not mean taht readers know it if it is not part of some other document
that is a reasonable context. (The OSPF base document is reasonable
assumed context for an OSPF extension. But the full set of all
published BGP extensions that are not mentioned in the document is not
suitable context for a BGP extensions. Fortunately, the recent
extension I reviewed did state what document was needed for context.)
If the reviewer is confused, it is reasoanble to assume that other
readers will be confused.
On design decisions, there is an evenr more complex tradeoff. I have
reviewed several documents which had questionable design decisions. In
one review I recently wrote that I did not expect the following comment
to change the WG consensus, but that I considered the specific mechanism
used by the document a bad idea. If I had not known that the particular
mechanism had been discussed, I might have put it more forcefully.
On the other hand, a while back I reviewed a document which had a
mechanism which, although the working group had indeed agreed on it,
fundamentally didn't work. I don't care how much they agreed. It
needed to be changed. And they changed it. (It was incumbent upon me
to provide a clear and coherent explanation of why it was broken.)
The problem I see is that working groups make all sorts of interesting
mistakes. Sometimes deliberately, sometimes by accident. The job of
the external reviewer is to look for such things.
Given that we have a policy that all comments deserve responses, it
seems particularly appropriate that review team comments better get
responses.
Also, on the time frames, it is worth noting that these are late
reviews. If you are lucky, they are produced early in the LC cycle.
But even reviewers are human and take some time. But if the review is
not responded to in a timely fashion, then other folks are likely to
assume that there is a real problem that needs to be addressed, rather
than finding out that the reviewer simply misread the document
(reviewers make mistakes too.)
The whole tone of your comment seems to be "these should have been made
during and as part of the WG process. But late external reviews benefit
from being external. (When I write a paper, I like outside review
because I get used to my own writing and assume that things flow.
Sometimes they don't Working groups make the same mistakes sometimes.)
Yours,
Joel M. Halpern
Ted Hardie wrote:
At 3:04 PM -0700 5/29/08, Suresh Krishnan wrote:
Hi Folks,
We have written a draft describing some guidelines for authors and
reviewers of internet drafts. We would really appreciate it if you can
spend some time to go over it and provide comments and/or suggestions
for improvement.
Thanks
Suresh, Pasi and Eric
Hi Suresh, Pasi, Eric,
I looked at it, and, while I laud any efforts to get folks to review
things effectively, I have to say that I found this to be a pretty drafty
draft.
It does not reference the Tao, 2026, or any of the developed educational
materials;
its only listed reference, in fact, is 2119 and that does not seem to be
referenced
within the text. That means that this comes across as pretty context free.
It needs anchoring to the rest of our processes.
One of the things that I believe that anchoring should provide
is a pretty significant change of perspective. As this reads now,
it implies a lot of power in the hands of reviews to elicit (or even require)
change. It seems to want document authors to accede to requests for
tutorial material as a matter of course and to significant technical changes
with a modicum of fuss. That's not the right approach. The outcome of our
document development should not be a negotiation between the authors
and the assigned reviewers. It should be a conversation in the working group
among those who will actually develop the implementations, those who
will deploy it, and those who are affected by the system of which the
documented technology is a part.
Reviews that work to relate a particular document's technology
to the larger whole of which it is a part (asking: how does this impact
congestion control in the access network or core, use deployed security
systems,
relate to the identifier mechanisms common to URIs, etc.) are very valuable.
But many reviews represent questions about decisions that come down to
design choices that working groups should have the power to make without
extensive second-guessing. Folks who want to have a say at the level can and
should do so with the simple method of joining the working group list and
commenting
as part of the general development. That's not "review" (of someone else's
work)
it is "participation" (in joint work), and it is fundamentally more valuable.
Without the context of how "participation" works, your documents description
of "review" comes off very badly indeed. I hope that future versions can
correct
that and place review within a broader, more productive context.
good luck,
Ted
-------- Original Message --------
Subject: I-D Action:draft-krishnan-review-process-00.txt
Date: Wed, 28 May 2008 11:15:01 -0700 (PDT)
From: Internet-Drafts(_at_)ietf(_dot_)org
Reply-To: internet-drafts(_at_)ietf(_dot_)org
To: i-d-announce(_at_)ietf(_dot_)org
A New Internet-Draft is available from the on-line Internet-Drafts
directories.
Title : Guidelines to authors and reviewers regarding the
IETF review process
Author(s) : S. Krishnan, et al.
Filename : draft-krishnan-review-process-00.txt
Pages : 10
Date : 2008-05-28
This document describes the IETF review process and provides
guidelines to draft authors and reviewers on how to effectively
participate in it.
A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-krishnan-review-process-00.txt
Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf