ietf
[Top] [All Lists]

Re: Alternate entry document model

2010-11-06 03:01:16
Hi,

I remember talking with Henning about the review process in the IETF
some time ago. I have pasted below the relevant section of his draft:

http://tools.ietf.org/html/draft-tschofenig-rai-reducing-delays-01#section-2

Review marathon:

      Getting a document through the IETF process does not happen
      without reviews.  There are reviews before the document becomes a
      WG item, reviews while the document is within the group, some
      chairs demand reviews before they issue a WGLC to probe
      'readiness', someone in between reviews by special expert groups
      are done (also by other SDOs and external groups, if necessary),
      then comes the WGLC, review by the responsible AD, review by the
      IESG, review by directorates (and there are many of those) and
      (depending on the type of document and history) an IETF Last Call.
      Everybody has an opinion, often not necessarily a technical
      opinion, on how documents should be written and why other solution
      approaches have not been explored.  Reviewers need time and then
      the review comments often cannot be ignored but need to be
      discussed and resolved.  When reviews happen later in the process
      then text changes are often expected to keep the reviewer happy.
      IESG members frequently put DISCUSSes on reviews and this
      increases their priority allowing a single person to, for example,
      delay the publication of a document for an extended period of
      time.  From a psychological point of view reviewers are in the
      unfortunate position that they have the feeling that something
      must be improved as an outcome of the review activity.  As soon as
      documents leave the working group the transparency is largely
      lost, despite IESG comments being sent to the authors, WG chairs
      and responsible ADs and despite information being available in the
      I-D tracker.  Mentally, many working group members consider
      documents to be 'done' when they leave the working group.

      The authors are not arguing that reviews are unnecessary but there
      has to be balance with respect to the goal that is about to be
      accomplished.


Note that this draft was written in the context of the RAI area but
seems to be equally applicable to other areas. Interestingly, the draft
also discussed the "Degeneration of the 3-level Standards Track process".

Cheers,

Gonzalo


On 31/10/2010 11:48 PM, Dave CROCKER wrote:


On 10/30/2010 11:31 AM, Joel M. Halpern wrote:
     One of the positive effects
of our current system is taht because WG knows tha tthye have to clear all 
the
ADs, not just their own, they actually think about all these issues. And 
usually
manage to cope with them

A working group is diligent or it isn't.  It gets a range of feedback and 
responds constructively to it... or it doesn't.

The working group behaviors that I have seen that pay explicit attention to 
the 
specific question of satisfying AD reviews have nothing to do with quality of 
the work and more to do with guessing what will personally bother an AD.  In 
other words, it's about dealing with AD idiosyncrasy rather than with quality.

It is now common to get cross-area reviews and my own observation is that 
these 
are a) typically quite reasonable and diligent, and b) dealt with 
constructive 
by the working group.

ADs do sometimes come up with interesting and even important points, but AD 
review is an extremely expensive and often frustrating mechanism that we 
already 
have a vastly superior replacement for.  Its timing is better and it 
distributes 
the work far better.

The fact that an AD sometimes catches some important problem is typically 
taken 
as proof that the AD review and Discuss mechanism is essential.  This is 
highly 
flawed logic, on two counts.

One is that it does not represent meaningful cost/benefit evaluation.  The 
cost 
is actually quite high in energy, delay and frustration, and the significant 
benefit overall is quite low (if the wg has been diligent and has gotten 
cross-area reviews.)

The other is that protocol specs have a statistical likelihood of bugs, even 
with the AD review.  We talk about AD review almost as if it ensures 
perfection, 
but of course we know it does not.

Ultimately, we have to trust the real world to evaluate the safety and 
efficacy 
of a protocol.  That fact ought to give us permission to balance the cost and 
benefit of the quality assurance efforts we require during specification 
development and approval.


Yes, it would be very good to spot all of these things sooner. I have not yet
seen a proposal that actually works for doing so. But letting WGs or WGs + 
ADs
approve documents for general advancement is a step likely to lead to 
problems.
If all our WGs handed their ADs high quality documents that they had checked 
for
all these issues, then maybe we could look at this differently.

We do need quality assurance efforts.  The basic idea that working group 
efforts 
are subject to outside review prior to approval is a significant value-add by 
the IETF, IMO.  The question is how to provide sufficient review in a 
reasonable 
way.

I believe that cross-area reviews largely satisfy that requirement.  If 
within-area reviews are also needed, the AD should commission them, not do 
them 
directly.

d/

_______________________________________________
Ietf mailing list
Ietf(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf

<Prev in Thread] Current Thread [Next in Thread>
  • Re: Alternate entry document model, Gonzalo Camarillo <=