Hi,
I remember talking with Henning about the review process in the IETF
some time ago. I have pasted below the relevant section of his draft:
http://tools.ietf.org/html/draft-tschofenig-rai-reducing-delays-01#section-2
Review marathon:
Getting a document through the IETF process does not happen
without reviews. There are reviews before the document becomes a
WG item, reviews while the document is within the group, some
chairs demand reviews before they issue a WGLC to probe
'readiness', someone in between reviews by special expert groups
are done (also by other SDOs and external groups, if necessary),
then comes the WGLC, review by the responsible AD, review by the
IESG, review by directorates (and there are many of those) and
(depending on the type of document and history) an IETF Last Call.
Everybody has an opinion, often not necessarily a technical
opinion, on how documents should be written and why other solution
approaches have not been explored. Reviewers need time and then
the review comments often cannot be ignored but need to be
discussed and resolved. When reviews happen later in the process
then text changes are often expected to keep the reviewer happy.
IESG members frequently put DISCUSSes on reviews and this
increases their priority allowing a single person to, for example,
delay the publication of a document for an extended period of
time. From a psychological point of view reviewers are in the
unfortunate position that they have the feeling that something
must be improved as an outcome of the review activity. As soon as
documents leave the working group the transparency is largely
lost, despite IESG comments being sent to the authors, WG chairs
and responsible ADs and despite information being available in the
I-D tracker. Mentally, many working group members consider
documents to be 'done' when they leave the working group.
The authors are not arguing that reviews are unnecessary but there
has to be balance with respect to the goal that is about to be
accomplished.
Note that this draft was written in the context of the RAI area but
seems to be equally applicable to other areas. Interestingly, the draft
also discussed the "Degeneration of the 3-level Standards Track process".
Cheers,
Gonzalo
On 31/10/2010 11:48 PM, Dave CROCKER wrote:
On 10/30/2010 11:31 AM, Joel M. Halpern wrote:
One of the positive effects
of our current system is taht because WG knows tha tthye have to clear all
the
ADs, not just their own, they actually think about all these issues. And
usually
manage to cope with them
A working group is diligent or it isn't. It gets a range of feedback and
responds constructively to it... or it doesn't.
The working group behaviors that I have seen that pay explicit attention to
the
specific question of satisfying AD reviews have nothing to do with quality of
the work and more to do with guessing what will personally bother an AD. In
other words, it's about dealing with AD idiosyncrasy rather than with quality.
It is now common to get cross-area reviews and my own observation is that
these
are a) typically quite reasonable and diligent, and b) dealt with
constructive
by the working group.
ADs do sometimes come up with interesting and even important points, but AD
review is an extremely expensive and often frustrating mechanism that we
already
have a vastly superior replacement for. Its timing is better and it
distributes
the work far better.
The fact that an AD sometimes catches some important problem is typically
taken
as proof that the AD review and Discuss mechanism is essential. This is
highly
flawed logic, on two counts.
One is that it does not represent meaningful cost/benefit evaluation. The
cost
is actually quite high in energy, delay and frustration, and the significant
benefit overall is quite low (if the wg has been diligent and has gotten
cross-area reviews.)
The other is that protocol specs have a statistical likelihood of bugs, even
with the AD review. We talk about AD review almost as if it ensures
perfection,
but of course we know it does not.
Ultimately, we have to trust the real world to evaluate the safety and
efficacy
of a protocol. That fact ought to give us permission to balance the cost and
benefit of the quality assurance efforts we require during specification
development and approval.
Yes, it would be very good to spot all of these things sooner. I have not yet
seen a proposal that actually works for doing so. But letting WGs or WGs +
ADs
approve documents for general advancement is a step likely to lead to
problems.
If all our WGs handed their ADs high quality documents that they had checked
for
all these issues, then maybe we could look at this differently.
We do need quality assurance efforts. The basic idea that working group
efforts
are subject to outside review prior to approval is a significant value-add by
the IETF, IMO. The question is how to provide sufficient review in a
reasonable
way.
I believe that cross-area reviews largely satisfy that requirement. If
within-area reviews are also needed, the AD should commission them, not do
them
directly.
d/
_______________________________________________
Ietf mailing list
Ietf(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf