Re: Work effort? (Re: Proposed Standard and Perfection)

--On Friday, 12 March, 2004 20:19 -0500 Keith Moore 
<moore(_at_)cs(_dot_)utk(_dot_)edu> wrote:
> >         (2) When a document comes up for review for
> >         Proposed->Draft, we look for implementations, etc.,
> >         perhaps following Keith's proposal outline.  If the
> >         implementations are there, we issue a Last Call for
> >         identification of serious technical/definitional flaws
> >         in the document as published, where "serious" is defined
> >         as "problematic enough to interfere with independent
> >         interoperable implementations".  If none are found, we
> >         advance the maturity level of the document, in place --
> >         no new RFC publication required and hence little or no
> > ...
> I cannot support the notion that the only technical flaws that
> should prevent a document from advancing from Proposed to
> Draft are those that "interfere with independent interoperable
> implementations".
>
> A protocol may interoperate quite well with itself, and yet
> have a peripheral effect on a large number of other protocols.
> A protocol that does not share channel capacity fairly with
> TCP can have a disruptive effect on TCP-based protocols.  A
> protocol that defines a new kind of (IPv4 or IPv6) address
> space that behaves differently from existing address space can
> break applications even if that protocol does not directly
> interact with those applications.  A protocol which doesn't
> provide adequate authentication can cause its hosts to be
> compromised which in turn affects the operation and security
> of other services.
> ...
Keith,

You are quite correct.  I oversimplified the situation 
considerably.  Clearly, any significant technical or 
interoperability difficulty that shows up should be considered 
and should be a showstopper for something going to Draft.  But 
consider:
(1) In your example above, I suggest that, with the amount to 
scrutiny now going into getting things to Proposed, the types of 
problems you identify above would be detected there and dealt 
with.  And, if they were not, we have a more serious problem 
than criteria for draft (in today's world, not an ideal one).
(2) We need to look at the system as it exists, not in how 
things might work if, e.g., everything got the perfect levels of 
review at the perfect time.  If we do that, our situation today 
is that documents go to Proposed, with whatever quality of 
(typically pre-implementation) review they get.  They get 
published.  Now, consider what might happen next, in an 
environment in which, for whatever reason (and I think both your 
observation about implementation reports and mine about 
document-fussing and general pain are part of the problem), we 
move almost nothing to Draft...
        * The Proposed Standard is implemented, causes no
        problems, turns out to be clear.  No one cares about
        moving it to Draft, so it stays at Proposed.
        
        * Flaws are discovered.  Our only real mechanism for
        documenting them is to reissue at Proposed or move the
        document to Draft (the RFC Editor's Errata are not
        IETF-authoritative and few people know where to find
        them).  But nothing goes to Draft, so the document sits
        there at Proposed, warts and all.
        
        * There are some other cases, but few result in the
        documents getting recycled, and fewer result in them
        going to Draft.

Not a good situation.  In fact, I think it is a sufficiently bad 
one that we should be looking at ways that would make things 
even slightly better in even a subset of cases than figuring 
what problems those changes would not fix in a perfect world.
> That, and it's too easy for technical flaws to be missed at
> Proposed Standard that show up after some implementation
> and/or deployment experience is gained - and these aren't at
> all limited to flaws that interfere with interoperation.
Sure.  Do you have a proposal as to how to get those flaws 
documented and the documents appropriately reissued?  If not, 
this is an "IETF is going downhill, news at 11" sort of 
observation.
> I do agree that we should not reopen noncontroversial
> documents that aren't found to have significant flaws.  I have
> occasionally suggested that rather than re-edit documents
> advancing in grade, we start out by making lists of things
> that need to be changed.  If that list ends up being less than
> N pages or X% of the length of the original specification,
> publish the list with a preface that says "RFC XXXX, with the
> following clarifications, changes, and corrections, is
> approved for Draft Standard".
We are in general agreement about this, and it is part of what 
my "turn STDs into real documents" idea was intended to address.
> OTOH, some documents that were controversial at Proposed
> probably do deserve special scrutiny at Draft.  That doesn't
> mean that we should automatically reopen them, but it might be
> that given the subsequent experience with the document it no
> longer meets the criteria for Proposed (e.g. "no known
> technical omissions").
Yes.  I was concerned about the ones about which there was no 
_real_ controversy (i.e., at the level at which a WG would have 
rough consensus on their being a controversy) but a large amount 
of noise... and no new facts arising during the interim period.
> > Principle: If we are going to spend as much time and energy
> > as we do  getting something to Proposed, then moving it to
> > Draft or Full should  usually require only identification of
> > implementations, deployment,  and desirability, not extensive
> > and time-consuming document rewriting
> What if we applied the same "principle" to software?  What if
> once a program were written there were an expectation that it
> should never be substantially changed, never be re-evaluated
> in light of changed conditions or new understanding?
Well, this isn't software.   And, if there is the incentive and 
energy to do more, I'm all in favor of it.   But I'm concerned 
about the cases where there isn't energy to do a lot more work 
but the implemented and deployed status of a specification (with 
or without some tuning and additional explanation) justifies its 
promotion.  If we can't solve that problem, discussions about 
retiring un-promoted standards will either turn out to be 
pointless or a mechanism for getting our decisions ignored in 
the marketplace.
Harald has pointed out that much of this is being discussed in 
newtrk.  I suggest that we move the discussion there if it is 
worth any further effort.
   john