On 5/30/08 7:54 PM, Ted Hardie allegedly wrote:
There are several issues that get mixed up together in defining a late
external review process. By definition, this is an external review. So
it is not reasonable to say that the reviewer should think like a WG
member, or have the full email conversations available.
No, it is not. But it is also not reasonable to assume that each document
carry the full context of an area with it. There is a balance there,
and assuming that contextualizing information should be added to
each document on the basis of reviewer request is a problem. In addition
to making each document very large, they run the great risk of getting
out of synch, with the usual hilarity resulting.
If there is _written_ context in other drafts, that's fine. If
_cultural_ context is missing from the drafts, that's a problem that
should be noted and fixed. Sometimes the drafts where the context
should be added are already out the door, and the only place to put the
needed context is in the draft under review.
But we're not writing our documents for reviewers; we're writing them for
implementors, operators, and our successors, toiling in these protocol farms,
right? So there is some level of common knowledge among those working
within a protocol area we can and should assume in our documents.
You can't assume that. We get strange questions from implementors and
operators all the time. Far stranger than the ones from reviewers.
Better to assume that reviewers understand the audience as well as you
do and what their needs will be.
We can and should provide citations (that's what those nifty normative
and informative references are for), but that doesn't translate into providing
tutorial text. I was once a big fan of it, but I see people go wrong on the
basis
of thrown-in tutorial text often enough now that I think it is often
less than helpful and sometimes actively harmful (by causing the reader
to believe that it is sufficient context).
Personally I agree in separating out tutorial text, but all context
required must be supplied somewhere.
On 6/2/08 3:00 PM, Ted Hardie allegedly wrote:
A review that is raising a showstopper has a provable or disprovable statement
in it. "This *will not work* in the following scenario" or even "This seems
to have poor results in networks with high rates of non-congestive loss"
creates the
opportunity for the reviewer and working group to discuss the issue in terms
that can be tested.
Agreed, and reviewers with expertise do this. Reviewers with less
expertise may just explain their doubt and ask if it is justified, but
even questions like that should be considered if they have not been
already, because on occasion I've seen them save a group from disaster.
But review comments that do not contain testable assertions end up
being subjective. "I think ZNK ChillOut is better than ZNK BindMeTight,
for the following reasons" may contain a good set of reasons, but it
should never over-rule the consensus of a working group that has
agreed on ZNK BindMeTight unless one of those reasons amounts to
"it doesn't work".
In many WGs there are multiple designs that would work, and various
tradeoffs are made in choosing one. There is never (afaik) a
requirement that one approach prove the other simply does not work.
Similarly early reviews can be very helpful even in re-examining known
tradeoffs or bringing up new ones so that the balance between the
approaches changes. It is rare for a late review to change a WG
decision in this way -- it's hard for a WG to backtrack -- but it does
happen occasionally, and there is never a requirement that a review
prove that a design is utterly broken, just that there are serious
disadvantages.
If a reviewer comment is opinion based on a great deal of experience,
for example with system design, then the reviewer -- presenting
him/herself as having special expertise -- then has an obligation to
recommend a solution to the problem he/she sees, in detail.
Comments of the type "I think Section 4 is not
clear enough for an implementor to follow" are also subjective; they
are very valuable and may serve as a guide to the WG/author team,
but it is important for the reviewer to recognize that they may not
result in change.
Yes.
Scott
_______________________________________________
IETF mailing list
IETF(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf