ietf
[Top] [All Lists]

Re: Guidelines for authors and reviewers

2008-05-30 16:58:37
At 4:08 PM -0700 5/30/08, Joel M. Halpern wrote:
I have to strongly disagree with the tone of this review of the review
document.
(Which is not to say that Ted isn't entitled to provide the review, or
that he is not entitled to a reasonable response.)

Tone is notoriously hard to get right, in email especially.  As an
early reviewer here, I am trying to make a forceful statement that
this is off on the wrong foot and needs to start over to make much
progress.  If that force comes off as rude, forgive me.  But this
document lacks context that you have at this point absorbed into
your bones, and I believe that without it, it will be harmful.


There are several issues that get mixed up together in defining a late
external review process.  By definition, this is an external review.  So
it is not reasonable to say that the reviewer should think like a WG
member, or have the full email conversations available.

No, it is not.  But it is also not reasonable to assume that each document
carry the full context of an area with it.  There is a balance there,
and assuming that contextualizing information should be added to
each document on the basis of reviewer request is a problem.  In addition
to making each document very large, they run the great risk of getting
out of synch, with the usual hilarity resulting.

So, the first question is what should happen when a reviewer says "X
does not make sense" or "Y seems to be assumed without explanation."
Some of the time, that is because that context is provided by other
documents which are reasonable to have read.  Someone needs to say that.
 (If a reviewer has become confused, likely some AD sill also become
confused.  Maybe the sponsoring AD will realize that the solution is
some other document that is already referenced.  But then again, maybe
he won't.)

An awful lot of the time though, the issue is one of "common knowledge".
 Sometimes bluntly stated as "all the participants understand that Z,
which means ..."  Well, the fact that working group members know Z does
not mean taht readers know it if it is not part of some other document
that is a reasonable context.  (The OSPF base document is reasonable
assumed context for an OSPF extension.  But the full set of all
published BGP extensions that are not mentioned in the document is not
suitable context for a BGP extensions.  Fortunately, the recent
extension I reviewed did state what document was needed for context.)
If the reviewer is confused, it is reasoanble to assume that other
readers will be confused.

But we're not writing our documents for reviewers; we're writing them for
implementors, operators, and our successors, toiling in these protocol farms,
right?  So there is some level of common knowledge among those working
within a protocol area we can and should assume in our documents.
We can and should provide citations (that's what those nifty normative
and informative references are for), but that doesn't translate into providing
tutorial text.   I was once a big fan of it, but I see people go wrong on the 
basis
of thrown-in tutorial text often enough now that I think it is often
less than helpful and sometimes actively harmful (by causing the reader
to believe that it is sufficient context).



On design decisions, there is an even more complex tradeoff.  I have
reviewed several documents which had questionable design decisions.  In
one review I recently wrote that I did not expect the following comment
to change the WG consensus, but that I considered the specific mechanism
used by the document a bad idea.  If I had not known that the particular
mechanism had been discussed, I might have put it more forcefully.
On the other hand, a while back I reviewed a document which had a
mechanism which, although the working group had indeed agreed on it,
fundamentally didn't work.  I don't care how much they agreed.  It
needed to be changed.  And they changed it.  (It was incumbent upon me
to provide a clear and coherent explanation of why it was broken.)

These both sound like excellent reviews:  you expressed your personal
design preferences in the first instance but did not try to force it over
the consensus of the working group, and pointed out a showstopper
in the second. 

Now, show me in this draft how these two cases are distinguished,
which is critical to getting a review done right? 


The problem I see is that working groups make all sorts of interesting
mistakes.  Sometimes deliberately, sometimes by accident.  The job of
the external reviewer is to look for such things.

Mistakes are not "does not agree with my prejudices", and that is a key
problem in many of the reviews we actually see. 



Given that we have a policy that all comments deserve responses, it
seems particularly appropriate that review team comments better get
responses.

Uh, why?  If you up and decide to read every 17th Last Call for your
own edification and respond with comments  that are reasonable reviews,
why should someone else's comments get better responses than yours?



Also, on the time frames, it is worth noting that these are late
reviews.  If you are lucky, they are produced early in the LC cycle.
But even reviewers are human and take some time.  But if the review is
not responded to in a timely fashion, then other folks are likely to
assume that there is a real problem that needs to be addressed, rather
than finding out that the reviewer simply misread the document
(reviewers make mistakes too.)

The whole tone of your comment seems to be "these should have been made
during and as part of the WG process.  But late external reviews benefit
from being external.  (When I write a paper, I like outside review
because I get used to my own writing and assume that things flow.
Sometimes they don't   Working groups make the same mistakes sometimes.)

My comment is not simply that late reviews should be made earlier.  It is that
we need to distinguish between issues that a reviewer can and should raise
as showstoppers "Will not work| Will break Internet | Uses component parts in
ways will cause whole to fall apart" and those that question the decisions
of the working group when none of those conditions are fulfilled.  That kind
of issue has a time and a place; document development (and, if necessary, the
bis or ter versions) is where that should happen.  If you push for ZNK
ChillOut over BindMeTight in a late review, in other words, you need to show
that BindMeTight doesn't work.  If you push it over the consensus of the working
group at late stage, you should be satisfied with a "considered; choice made"
and nothing else; anything else puts the power for determining the shape of
the document in the hands of those doing the review instead of doing the work.
That's wrong, that's dangerous, and any document that seems to push that
point of view needs to be extensively revised or dropped.

To help with the tone issue, permit me to wish you a happy weekend,
                                regards,
                                        Ted


Yours,
Joel M. Halpern

Ted Hardie wrote:
At 3:04 PM -0700 5/29/08, Suresh Krishnan wrote:
Hi Folks,
  We have written a draft describing some guidelines for authors and
reviewers of internet drafts. We would really appreciate it if you can
spend some time to go over it and provide comments and/or suggestions
for improvement.

Thanks
Suresh, Pasi and Eric

Hi Suresh, Pasi, Eric,

      I looked at it, and, while I laud any efforts to get folks to review
things effectively, I have to say that I found this to be a pretty drafty 
draft.
It does not reference the Tao, 2026, or any of the developed educational 
materials;
its only listed reference, in fact, is 2119 and that does not seem to be 
referenced
within the text.  That means that this comes across as pretty context free.
It needs anchoring to the rest of our processes.
      One of the things that I believe that anchoring should provide
is a pretty significant change of perspective.  As this reads now,
it implies a lot of power in the hands of reviews to elicit (or even 
require)
change.   It seems to want document authors to accede to requests for
tutorial material as a matter of course and to significant technical changes
with a modicum of fuss.  That's not the right approach.  The outcome of our
document development should not be a negotiation between the authors
and the assigned reviewers.  It should be a conversation in the working group
among those who will actually develop the implementations, those who
will deploy it, and those who are affected by the system of which the
documented technology  is a part.
      Reviews that work to relate a particular document's technology
to the larger whole of which it is a part (asking: how does this impact
congestion control in the access network or core, use deployed security 
systems,
relate to the identifier mechanisms common to URIs, etc.) are very valuable.
But many reviews represent questions about decisions that come down to
design choices that working groups should have the power to make without
extensive second-guessing.  Folks who want to have a say at the level can and
should do so with the simple method of joining the working group list and 
commenting
as part of the general development.  That's not "review" (of someone else's 
work)
it is "participation" (in joint work), and it is fundamentally more valuable.
Without the context of how "participation" works, your documents description
of "review" comes off very badly indeed.  I hope that future versions can 
correct
that and place review within a broader, more productive context.
                      good luck,
                              Ted










-------- Original Message --------
Subject: I-D Action:draft-krishnan-review-process-00.txt
Date: Wed, 28 May 2008 11:15:01 -0700 (PDT)
From: Internet-Drafts(_at_)ietf(_dot_)org
Reply-To: internet-drafts(_at_)ietf(_dot_)org
To: i-d-announce(_at_)ietf(_dot_)org

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

       Title           : Guidelines to authors and reviewers regarding the
IETF review process
       Author(s)       : S. Krishnan, et al.
       Filename        : draft-krishnan-review-process-00.txt
       Pages           : 10
       Date            : 2008-05-28

This document describes the IETF review process and provides
guidelines to draft authors and reviewers on how to effectively
participate in it.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-krishnan-review-process-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf

_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf


_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf