ietf
[Top] [All Lists]

draft-housley-two-maturity-levels

2010-07-04 19:39:15
Russ and other IESG members,

I have been trying to avoid getting embroiled in IETF process
issues since the collapse of NEWTRK, but, after reviewing
draft-housley-two-maturity-levels and the discussion on the
IETF list, the situation leaves me little choice other than
reengaging.  Several of the comments that follow overlap
comments others have made on the list, others are new (or
updated versions of ones that predate the current proposals),
but I think it is worthwhile to try to draw things together.

This (unfortunately rather long) note addresses several topics
that are almost, but not quite, separable.  If people want to
address specific ones of them, it may be useful to change
titles and open up separate threads.

They are:

(1) Analysis of the problem as stated in
draft-housley-two-maturity-levels-01

(2) Promoting more use of Draft Standard (under any name) by
eliminating Full Standard.

(3) Conflating protocol quality with document quality

(4) Downward references

(5) Abandonment of recommendation levels

(6) The futility of using a small number of categories to
describe a multidimensional situation: Status description
versus status categories

(7) The process questions surrounding proposal, posting, and
processing of this document.  

  -----------------------------

(1) Analysis of the problem as stated in
draft-housley-two-maturity-levels-01

The draft indicates (Section 1) that:

        "The proposed changes are designed to simplify the process
          and reduce impediments to standards progression"

and then

        "Since many documents are published as Proposed Standard
           and never advances (sic) to a higher maturity level, the
          initial publication receives much more scrutiny that is
          call (sic) for by RFC 2026 [1]."

However, if the issue is that documents rarely progress beyond
Proposed, then there is no evidence that tampering with the
relationship between the current Draft and Full Standard levels
will bring about any change in that relationship, i.e.,
increase the number of documents that move past Proposed.  I
note, as others have noted, that the current de facto
requirements for Proposed are far in excess of what 2026
requires.  If the problem is (as I believe it is at least in
part) that initial publication receives far too much scrutiny
and takes far too much time and energy, then formally no
procedural changes at all are needed: the IESG simply needs to
stop doing that and instead and apply only the level of
scrutiny that RFC 2026 requires, and no more.  Such a change
would result in more rapid publication of protocol documents as
Proposed Standards (perhaps reducing the fraction of the
Internet that actually runs on I-Ds) so the community can
evaluate whether reducing the community exhaustion from getting
things to Proposed results in more documents moving to the next
level (whether that is one next level or one of two.

It appears to me that there are portions of the IETF community
who are willing to advance documents to Draft.  Most of that
subcommunity is willing to put in the work to continue to
advance documents to Full Standard and sees value in doing so.
Other portions of the community see little value in moving
documents past Proposed.  It doesn't seem to me that this
proposal will benefit either sub-community -- those who won't
advance past Proposed under either system and those who are
willing to advance to Draft and Full-- so I'm not quite sure
who would benefit.

Those who have been anxious to advance documents within the
existing model have often met significant resistance from IESG
members.  It is somewhat disingeneous for the IESG (or even
some of its members) to resist efforts to move documents to
Draft or Full Standard, to place requirements on approval of
Proposed Standards that go well beyond those of 2026, and then
to complain about how few documents advance to Draft or Full
Standard and use those complaints as the justification for
abolishing the Full Standard level.

On the other hand, the analysis in the document ignores the
observation that, independent of the interoperability
demonstration, moving a specification up in maturity levels
often results in significant improvement in document quality
and clarity.  Again, it might represent an even larger
improvement if the IESG more closely adhered to what many of us
believe is the intent described in RFC 2026 and earlier
documents -- to get a specification published for examination
and implementation purposes as soon as all known technical
defects have been identified or eliminated but without
investing the resources to polish the text editorially.  See
(3) below for more discussion on that subject.

Conclusion: Whatever this proposal may or may not accomplish,
there is no evidence that it will solve the problems identified
in its Introduction.


(2) Promoting more use of Draft Standard (under any name) by
eliminating Full Standard and renaming.

YMMD, but I believe that the fraction of the community who now
believe that having something at Proposed (1st step) only is a
bar to implementation is fairly small.  If it weren't, we'd be
in bad trouble as a consequences of the very active use of a
number of protocols that have never advanced beyond Proposed.
It is possible --although I have some doubts-- that rebranding
would help here.  But, to the extent to which the need is to
rebrand, that does not, in itself, justify changing the
maturity level model.  

If one simply wanted to rebrand to reflect present-day reality,
we would change Proposed/ Draft/ Standard to Standard/ Standard
with Gold Star/ Standard with Gold Star and Network Cable
Wreath.  Those titles are somewhat in jest, but the point is
that one wants to make the first level "Standard" and then
start talking about endorsements of higher specification
quality or assurance (see #6 below).  

Our real terminology/ branding problem is that "Proposed
Standard" has periodically been interpreted -- sometimes by
people or organizations with other agendas -- as "substandard
and really not suitable for deployment".  In fairness to them,
that was exactly our intention in the early 1990s.  That
intention was very much part of the "upgrade or die" language
in 2026, but has never been used.  Leaving "Proposed Standard"
as part of our vocabulary doesn't solve that problem.

Finally, along the same lines, I disagree that STD numbers
"offer little value".  If properly used, they offer
considerable value in providing a stable identifier, at least
unless more sophisticated arrangements are adopted (see #6).
The problem is that we are largely running on Proposed
Standards and STD numbers are assigned only to Full Standards.
The NEWTRK WG discussed solving that narrow problem in early
2006 by simply assigning protocol-specific identifiers to
documents when they reached the first standards-track level.
Like most other NEWTRK proposals, it was never formally
considered by the community.  It would make much more sense to
follow that advice than to discard a facility that should be a
useful one.  See
https://datatracker.ietf.org/doc/draft-ietf-newtrk-docid/ for
more information and discussion.  I also note that BCP numbers
are sometimes used to cluster groups of documents just as STD
numbers are. In both cases, there may be issues with idea and
mechanism for clustering, but, if one can extrapolate from the
BCP experience, the problem with STD numbers is that they are
not assigned early enough, not that they exist and are
inherently confusing, as your draft suggests.

Conclusion: there is probably more than adequate justification
to do some rebranding and/or to assign STD numbers (or
equivalent) at entry into the Standards track, but the new
naming categories proposed will lose information without really
improving anything.


(3) Conflating protocol quality with document quality

Advancing documents according to the current model actually
provides two separate (and separable) functions.  One is
certification of some minimal proof of interoperability (Draft)
and utility (Full).  The other is the specifications almost
always undergo significant improvements in the quality of their
documentation and description as people are required to
re-examine and re-edit them in the process of advancing between
grades.  I suggest that the IESG is not taking enough advantage
of that distinction and the advantages of the review process
and may have lost sight of it entirely.  First, we would be
much better off in terms of the perceived speed with which we
can do work if we stopped doing editorial nit-picking of
first-level (Proposed Standard) documents, concentrating on
making them only good enough to be clear to a well-intentioned
reader.  Such nit-picking and fine-detail editorial work for
first-level documents is problematic whether it is done in the
WG or the IESG and almost as problematic if done by
professional RFC Editor staff.  If nothing else, it is one of
the factors that induces exhaustion with documents and acts as
a deterrent to advancing them.

Either way, we should have a high document-quality expectation
at the second (and, if it exists, third) maturity levels.
Pragmatically, that expectation is generally inconsistent with
either "adopt the first written version of the spec initially
at the highest level" or waving of the minimum time in grade
rules.  For the latter, some time away from the document and
opportunity for the community to try to use it are likely
prerequisites to having a useful editing/ document quality
improvement effort.  I would favor giving the IESG the ability,
after Last Call and demonstration of community consensus to
waive those time requirements in exceptional circumstances.
But abolishing them seems to me to be the wrong approach. 

Conclusion: Improving document quality of mature specifications
should be, and remain, an important goal.  An excessively high
threshold for Proposed Standards and reducing the number levels
and qualifications (including elimination of minimum times in
grade) for advancement may frustrate, rather than further, that
goal.


(4) Downward references

I don't see the evidence for "major cause of stagnation in the
advancement of documents" unless one relaxes the rule to permit
normative references to I-Ds (which I do not favor).  If the
IESG believes that a document is being stuck unreasonably (or
is likely to get stuck), it has the ability to invoke the RFC
4897, which merely requires asking the community (in a Last
Call that can be combined with the Last Call on the document if
desired but that can also be applied later) and noting the
downref in the document.  That is not an onerous procedure, yet
the IESG has never chosen to use it.  If we really want to
treat standards at the second maturity level (or third, if it
remains) as better and more completely-specified than those at
lower levels then the downref restriction should remain, with
the community able to make exceptions when the lower-level
document(s) are considered sufficiently mature and
well-specified.  If, by contrast, additional maturity levels
are endorsements indicating that almost-orthogonal quality
levels have reached (as in the "gold star" recommendation of #2
above), then the downref issue may become irrelevant.

Conclusion: This proposed change solves a problem that has not
been demonstrated to exist, even to the extent of demonstrating
that the elimination of current mechanisms for permitting
downward references would save the IESG significant time.


(5) Abandonment of recommendation levels

Section 3.3 of RFC 2026 specifies a set of "requirement levels"
for technical specifications (usually supplied in an
applicability statement).  We have largely stopped using those
requirement levels, but the conflation of publication of the
document in the standards track (and some decisions to not
publish in the standards track) with recommendations that the
specification actually be implemented and used has been another
source of confusion including the publication of specifications
for some very stable and extensively deployed protocols and
other functions as Informational or Experimental, rather than
Standards Track ("if you are going to do this, this is the
interoperable way to do so") with "not recommended" ("but you
probably shouldn't do it at all").  Discussion of documents
that seem to require being misclassified is another impediment
to rapid and efficient progress.  It is disappointing that the
IESG did not decide to address this issue.

Conclusion: Whatever its other strengths and weaknesses, the
proposal is patchwork that ignores a major component of the
situation and should not be approved until and unless that
issue is addressed.  Suggestions on the IETF list, including,
if I recall, some from IESG members, that the proposal should
be approved in Maastricht are obviously inconsistent with the
nature of the proposal and its relationship to actual cause and
effect analysis.


(6) The futility of using a small number of categories to
describe a multidimensional situation: Status description
versus status categories.

It seems to me that much of the root problem is that we are
trying to use a small number of labels -- Proposed, Draft, and
Full in the current arrangement --to describe some rather
complex categories.  We've seen extensive deployment and use of
Proposed Standards, some of which are rather badly documented.
We've seen specifications widely deployed but based on a single
reference implementation and therefore not meeting the
interoperability tests for Draft.  We've seen specifications
held up because of quibbles about interoperability testing in
spite of the fact that every participant in the IETF is using
products based on those protocols on an everyday basis.  And we
have seen a huge amount of confusion about the relationships
among various documents -- confusion that assignment of STD
numbers doesn't solve, but at which they might do better if we
assigned them earlier in the process (see #2 above).  

While it was widely misunderstood (and sometimes
misrepresented), the original intention behind the "ISD"
proposal was to gradually eliminate excessive reliance on
maturity level categories, requirements categories, and STD
numbers and the inherent rigidity of each in favor of textual
descriptions of what we know, what has been examined, and what
the relationships are.  In retrospect, some of the requirements
and mechanisms were probably applied/ required at the wrong
times and too generally: there are probably situations in which
nothing is needed, situations in which Doug's "cover sheet"
proposal or something very much like it would be sufficient,
and situations in which all or part of a full description is
more appropriate.  It is also worth noting in this context that
the errata mechanism, while better than nothing, is not
sufficient to solve any of the problems with the standards
track... if only because even an "approved" erratum isn't an
authoritative, community consensus document but merely a
prediction of the consensus the community might reach if asked.
An extension of the ISD model also provides a way for the
community to say, by the usual consensus mechanisms, that a
provision of a protocol spec is in error or ambiguous and that
it should be read in some particular way (or replaced with
particular text).

For discussion purposes, I'm going to try to get an updated
version of the ISD proposal --one that includes some of the
thinking referred to above-- posted before the cutoff.


(7) The process questions surrounding proposal, posting, and
processing of this document.  

Independent of the content of this document, the procedure used
to develop it, discuss it in the community, and presumably to
reach conclusions about it seems to be to be seriously
questionable.  It is identified as derived from an IESG
discusson.  It was written by the IETF Chair and General Area
Director, who has previously taken the position that process
change proposals were unwelcome unless they met narrow
categories, categories that were not widely exposed to the
community much less the result of community consensus.  It has
been scheduled for plenary time at IETF 78 (by the IETF Chair
and/or the IESG) and announced with timing that makes
preparation of well-developed alternate proposals difficult.  I
note that the IESG discussions that led to this proposal
apparently do not even appear in published minutes.  Even if
the IESG review is ultimately "sponsored" by some other AD, the
fact that this proposal affects the way the IESG does business
and is derived from IESG discussions gives it the appearance of
an IESG proposal, scheduled for discussion by the IESG (or an
IESG member) under conditions set by the IESG or that member,
and then subject to consensus review and approval by the same
IESG.  This does not appear to me to be how we want to do
things; certainly it calls the openness and transparency of the
IETF into question unless we argue that IESG members are
granted powers to make for the community and independent of any
requirement for fairly-determined community consensus by virtue
of appointment to their positions.

We ought to be able to figure out a better way.  Perhaps the
development and advocacy for this document even illustrates
that a different way to deal with process change proposals
should be the next change that should be made to 2026.

regards,
    john


_______________________________________________
Ietf mailing list
Ietf(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf

<Prev in Thread] Current Thread [Next in Thread>