ietf
[Top] [All Lists]

Re: Guidelines for authors and reviewers

2008-06-02 12:10:43
At 5:12 PM -0700 5/30/08, Joel M. Halpern wrote:

These both sound like excellent reviews:  you expressed your personal
design preferences in the first instance but did not try to force it over
the consensus of the working group, and pointed out a showstopper
in the second.

Now, show me in this draft how these two cases are distinguished,
which is critical to getting a review done right?

The problem I have is that I do not know how to write text in a draft
that distinguishes those two.  The line between them is very tricky, and
possibly subjective.

Here is where I think we may have some room for some fruitful discussion.

A review that is raising a showstopper has a provable or disprovable statement
in it.  "This *will not work* in the following scenario" or even "This seems
to have poor results in networks with high rates of non-congestive loss" 
creates the
opportunity for the reviewer and working group to discuss the issue in terms
that can be tested.   The working group may argue (and, presumably, demonstrate)
that either statement is false, but there is a way to move forward that is not
just iteration of position.  To put that task to the working group, a reviewer
should be able to back their statement with a reasonably clear, worked set of
reasons for the assertion.   A working group can disprove the statement,
indicate why an applicability statement is a better response ( demonstrating
that the cases where it does work or does have reasonable results are useful
enough and distinguishable enough that the document should go forward),
or make relevant technical changes.

But review comments that do not contain testable assertions end up
being subjective.  "I think ZNK  ChillOut is better than ZNK BindMeTight,
for the following reasons" may contain a good set of reasons, but it
should never over-rule the consensus of a working group that has
agreed on ZNK BindMeTight unless one of those reasons amounts to
"it doesn't work".  Comments of the type "I think Section 4 is not
clear enough for an implementor to follow" are also subjective; they
are very valuable and may serve as a guide to the WG/author team,
but it is important for the reviewer to recognize that they may not
result in change.  

And part of the problem is to avoid turning it into a fight.  If all
review comments get clear, reasonably timely responses, there is room
for the discussion without acrimony.

I don't think that helps unless there is a clear set of expectations
*on the part of reviewers* on what their responsibilities are in
producing an actionable review and in accepting that some of their
comments may result in no change.  I believe that the right way
to do that is to ground the description of the review process in
a strong understanding of the document production and participation
model of the IETF.  If you don't, you end up with an unbalanced
view in which the few hours of effort put in on a solicited review
have a very much larger effect than the effort of the participants
who produced the document and will use the protocol it
describes.

Would it address your concern if the document said something like:
   "Reviewers should be sensitive to the difference between
   their personal opinions (and preferences) and issues
   which will affect the correct operation or interoperation
   of the documents under review"
?

I don't think this is nearly enough.

                                        Ted


I have no problem with pointing out that there are two different
categories.  I have real problems with trying to define a hard line
which distinguishes them.

Yours,
Joel

PS: While there are other differences in our views, they seem to be
questions on which reasonable folk may differ and we can let the
community sort out how to write the wording.
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf

_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf