--On Saturday, 08 March, 2008 00:12 -0800 Lakshminath Dondeti
<ldondeti(_at_)qualcomm(_dot_)com> wrote:
The ideal way to deal with them is to always respond, and to
get an "I am satisfied with your response" to close the
thread.
"Ideal" being the keyword though. Not everyone, for any
number of reasons, including cultural reasons, will come out
and state "all clear." It is also asking too much to ask the
reviewer to get into a debate with the authors. It also
fosters an environment where the reviewer starts becoming an
authority.
...
Repeat above several times and intersperse with long periods
of time where nothing happens on a document. You now have an
idea of why it seems to take a long time to get documents
through the system.
Indeed. What started out as a great idea -- I volunteered to
be a GenART reviewer 3-4 years ago now -- is beginning to
become yet another burden in the process.
I have to agree with this. One of the biggest risks we have to
quality in standards is the dark side of the review process, a
situation in which the effort to get a document as nearly
correct as is reasonable given its maturity level and possible
turns into "deal with that objection". A half-dozen years ago
we had extensive discussions of a concern in which ADs would
object to particular text and authors, exhausted from the
process of getting documents produced, would agree to any
suggested changes --as long as they were not really offensive--
in order to make the objection go away. Put differently, there
is a tendency for "satisfy him (or her) and make the DISCUSS go
away" to become a more important objecting in practice than "get
things right".
In at least some ways, the DISCUSS criteria were an attempt to
constrain that problem, at as as far as the ADs were concerned.
If we replace the "opportunity" to have to individually satisfy
a dozen or so ADs with the opportunity to have to individually
satisfy them plus a dozen more area-related reviewers, we are in
big trouble.
I am greatly in favor of these invited reviews if they ensure
that every document is carefully reviewed by someone who was not
part of its development process. I think that is where we
started out.
But the IESG has been selected by the community to take
responsibility for these evaluations. If the IESG isn't going
to do it, or can't, we need to be looking at our basic processes
and structure, not at who is reviewing what for whom. If a
review is done for a particular area, I expect that review to be
treated primarily as advice to the relevant ADs. I expect those
ADs to evaluate that advice carefully, not just to critically
accept it.
I definitely do not want to see a discussion between authors and
reviewers --especially Area-selected reviewers-- during Last
Call. It too easily deteriorates into a "satisfy him"
situation, and those reviewers are not anything special (or,
unlike the ADs themselves, selected by some community mechanism).
I think this whole process needs a little refocusing.
Especially for WG documents, no matter how many reviewers,
shepherds, and lions, tigers, and bears we introduce into the
system, the IESG should have one primary focus, which is making
a "go / no-go" decision on whether a document is ready for
approval and publication given maturity levels and any other
relevant issues. If the answer is "no-go" for anything but an
obvious matter about which there is no dissent or ambiguity, the
document ought to go back to the WG for resolution of issues
(which should clearly be identified as clearly as possible), not
turned into a negotiation process between whomever happened to
generate a comment and the authors about whether that person's
view of the comment can be satisfied. There is just too much
risk in the "satisfy comments" model of getting something
important wrong or of responding to the comment but missing the
main point of which the comment is a symptom.
I certainly don't object to Thomas's idea of using issue
trackers more and I think that making reviews public is an
important safeguard. But those issue trackers should be used to
* inform the IESG's decision about whether the document
is ready and to
* help inform the WG, presumably along with a careful
IESG summary, about the issues (not the specific
comments) it needs to address if the document is bounced
back to it.
Except perhaps for editorial matters and for clarification, a
dialogue between an author and someone who makes a comment
should have no place in the consensus process.
john
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf