Trying to paraphrase Ted's response to my comments, and then reply.
(The paraphrase is to try to make sure that I understood the comment.)
I think that Ted is saying that review comments should have testable,
actionable comments. Further, if I am reading him right, he is saying
that comments that are not testable are probably opinion.
That does seem a good place to start. Further, Ted comments taht
reviewers should understand that not all their comments are going to be
accepted. I agree. Not even all of their comments based on proper
understanding of the document are going to be accepted.
But, the question of where the boundary is between "that does not work"
and "in my opinion that is a bad idea" is, in my experience, frequently
fuzzy. To use a concrete example, we have plenty of experience to show
that using a field length to determine what kind of field something has
is a bad idea. It fails, over time, in a myriad ways. But no matter
how many horror stories I put together, I was going to lose that
particular debate with that particular document.
On the other hand, there are things which look perfect in theory, but we
know turn out to be more fragile and are therefore bad ideas. The
assertion that such a thing is a bad idea is going to look like opinion,
but is useful.
In fact, part of my concern is that sometimes opinions can be as
important as facts. Yes, reviewers need to be prepared to accept
push-back. But it should not be against the rules for a reviewer to say
"I have seen exactly this before, and by the time they made it work it
was a mess. Please don't do this."
And even if it is opinion, it deserves a response.
TO use another example, when I said to a working group "Having worked
with a lot of testers, I think that they will have trouble following the
instructions as you have written them." I was clearly stating an
opinion. But it was an opinion that at least deserved a response. (In
fact, the working group did respond, and a good compromise was reached
on that and a number of related issues.)
Part of this is that just as we expect authors to respond politely to
reviewers, reviewers have an obligation of politeness towards the WG.
Reviewers should assume that odd things are there for good reasons, even
while raising the question as to whether they belong.
But trying to create a hard line between opinion and fact is I believe
counter-productive. If the issue is larger than it looks on reading the
review, then the reviewer can add more explanation, examples, or
justification in the dialog with the authors / WG. Similarly, if the WG
is overreacting to something but feels it is a big deal, the reviewer
can say in followup "I had not realized that would be a big deal. Let
it be."
We are all presumably adults. We are also human.
There is also a subtext that comes up periodically about the comments
being actionable. This has been raised for outside reviews and IESG
reviews. While I certainly agree that actionable comments are much
better, it is not always possible. If the problem is that the reader
can not figure out what the text means, it is hard to suggest a fix. It
also gets laborious if the text could mean one of three different
things. Should the reviewer have to provide text for all three
alternative meanings? After all "clarify the text" while clear is not
very specific as to what action is needed. There are also other cases
that come up where it may be impractical to require actionable comments.
Yours,
Joel M. Halpern
Ted Hardie wrote:
At 5:12 PM -0700 5/30/08, Joel M. Halpern wrote:
These both sound like excellent reviews: you expressed your personal
design preferences in the first instance but did not try to force it over
the consensus of the working group, and pointed out a showstopper
in the second.
Now, show me in this draft how these two cases are distinguished,
which is critical to getting a review done right?
The problem I have is that I do not know how to write text in a draft
that distinguishes those two. The line between them is very tricky, and
possibly subjective.
Here is where I think we may have some room for some fruitful discussion.
A review that is raising a showstopper has a provable or disprovable statement
in it. "This *will not work* in the following scenario" or even "This seems
to have poor results in networks with high rates of non-congestive loss"
creates the
opportunity for the reviewer and working group to discuss the issue in terms
that can be tested. The working group may argue (and, presumably,
demonstrate)
that either statement is false, but there is a way to move forward that is not
just iteration of position. To put that task to the working group, a reviewer
should be able to back their statement with a reasonably clear, worked set of
reasons for the assertion. A working group can disprove the statement,
indicate why an applicability statement is a better response ( demonstrating
that the cases where it does work or does have reasonable results are useful
enough and distinguishable enough that the document should go forward),
or make relevant technical changes.
But review comments that do not contain testable assertions end up
being subjective. "I think ZNK ChillOut is better than ZNK BindMeTight,
for the following reasons" may contain a good set of reasons, but it
should never over-rule the consensus of a working group that has
agreed on ZNK BindMeTight unless one of those reasons amounts to
"it doesn't work". Comments of the type "I think Section 4 is not
clear enough for an implementor to follow" are also subjective; they
are very valuable and may serve as a guide to the WG/author team,
but it is important for the reviewer to recognize that they may not
result in change.
And part of the problem is to avoid turning it into a fight. If all
review comments get clear, reasonably timely responses, there is room
for the discussion without acrimony.
I don't think that helps unless there is a clear set of expectations
*on the part of reviewers* on what their responsibilities are in
producing an actionable review and in accepting that some of their
comments may result in no change. I believe that the right way
to do that is to ground the description of the review process in
a strong understanding of the document production and participation
model of the IETF. If you don't, you end up with an unbalanced
view in which the few hours of effort put in on a solicited review
have a very much larger effect than the effort of the participants
who produced the document and will use the protocol it
describes.
Would it address your concern if the document said something like:
"Reviewers should be sensitive to the difference between
their personal opinions (and preferences) and issues
which will affect the correct operation or interoperation
of the documents under review"
?
I don't think this is nearly enough.
Ted
I have no problem with pointing out that there are two different
categories. I have real problems with trying to define a hard line
which distinguishes them.
Yours,
Joel
PS: While there are other differences in our views, they seem to be
questions on which reasonable folk may differ and we can let the
community sort out how to write the wording.
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf