Stephen,
Thanks for proposing this. You know that I agree with you that improving IETF's ability
to publish specifications relating to real code out there is important. It is important
to come up with ideas in this space. I think there has been situations where IETF has
strayed from the "rough consensus and running code" model.
But of course following the running code is not always easy. I have some
perspectives on this, both from the IESG side and as a document author who in
some cases had running code. Obviously we want IETF's resources spent on things
that result in running code, and we want experiences from that running code
reflected back into specifications. And we want those fast-running
implementation teams to bring their specs to the IETF rather than publish them
somewhere else.
But on the other side, there are issues. It is sometimes difficult to
distinguish useless junk with several implementations from useful and stable
technology that is going to change the Internet and which also has several
implementations. And it is difficult to guess whether an implementation is the
beginning of something great or the by-product of someone's pet project that
also produced the specs. In many cases it is very cheap to write an
implementation. You can also easily build an implementation and fund a few
other folks to build theirs. And even if you have independent implementations,
I've seen cases where there are seven implementations that inter-operate but
the spec is still unimplementable for outsiders.
If you want to support IETF work that has implementations better, you have to decide
where your focus is in the overall process. I suspect that the last calls make up a very
small part of IETF's overall end-to-end ("draft-00-to-RFC") delay. I do not
have good statistics of this at the moment,
http://www.arkko.com/tools/lifecycle/wgdocs.html has some numbers from buggy code that
has not been run since last spring. Those numbers seems to indicate WG process times in
the order of two years and LC/IESG times in the order of half a year, plus some RFC
editor time. Intuitively, the numbers match my own experience, so maybe they are in the
ballpark.
One of the issues is that it is very rare for document to sail through the
LC/IESG without any changes. From the same crappy statistics,
http://www.arkko.com/tools/admeasurements/stat/goodprocessingtime_4.html and
http://www.arkko.com/tools/admeasurements/stat/processingtime_4.html that's
maybe 10% of all documents; 20% in some cases but those drafts probably have
some RFC Editor notes associated with them. A document that does not change
goes through quite fast, maybe in a month. A document that needs change also
needs discussion, debate, and waiting for a revision. Hence the half a year.
Two weeks out of that? Not a big difference. Two weeks out of 2-3 years for the
entire spec development? Not noticeable.
All decisions involving how you give support or priority to running code
involve difficult judgment calls. There's running code. Does that trump
someone's opinion that the bits should be ordered differently? That the design
should be changed to work better through NATs? That no security is needed?
There is no hard and fast rule for these questions, you have to evaluate what
the situation is.
It is not easy to support running code as part of the IETF day-to-day process.
People test obviously, on their own. Many people bring implementations to IETF
meetings. Some organize back room meetings to do even proper interops. We saw
some code in bits-n-bytes, and there's been a few other demos as well. But
interops are not within IETF scope. It is even difficult to acquire resources
for testing in all cases. In Atlanta, we asked for a room the hallway was no
longer sufficient. We did this in the last minute, got some well-deserved
pushback from the NOC crew (who did a wonderful job and supported us anyway -
thanks). But we also got some unexpected pushback from the IESG, who among
other things told us that they can't help us with a room in any way until the
spec is officially a WG document. But to get to that state we'd need some
testing to improve the spec and convince people that the stuff works. So catch
22. Luckily, we got a room from IPSO Alliance, who stepped up to !
help.
Anyway, all this puts me roughly in the same camp as Barry. How about doing
this instead:
1) Write that IESG statement or new RFC that reminds everyone that they should
seriously consider what running code tells us when they weigh issues, decide to
send documents forward, etc. Obviously we should not follow code blindly.
Perhaps the statement could highlight some of the above issues, or the issues
from Stephen's draft to make people aware of the complexity of the matter.
2) Re-focus ourselves in the WG and IESG reviews on the stuff that actually
matters. DISCUSSes on what's in the document header of the I-D... do not matter
as much as whether the thing actually works, or is i18n capable or secure. Yes,
I know I've erred on this myself several times.
3) Remember that some of the most successful IETF specifications were things
that were brought to us from the outside, with implementations and first
standards revisions already done. It is not a bad model. You do not always have
to start with a committee. You can finish with a committee.
4) Find ways to support running code in all stages of the IETF process. Could
we support unofficial interops better in our meetings, or work with more
interop organizations? Could we use bits-n-bytes to do more demos, and would
this be useful in promoting running code?
Some more comments on details on your draft:
* The IPR return-to-WG process is actually part of existing operations for all
drafts. Perhaps you could clarify this.
* I'm not sure why you have taken the position that the sponsoring AD does not
have to vote Yes in these. I think that should be necessary even more than with
other drafts. Were you thinking that there's some other AD who knows the case
better? There are certainly cases where another AD knows more about a
particular case than the one officially in charge of the WG. In those cases it
might actually make sense for that other AD to take on the sponsoring role. But
that seems orthogonal to fast tracking, and is already happening.
* I think you have placed perhaps too much emphasis on verifying the software.
I think these are judgment calls, you figure out what is available, what its
functionality is, how it has been tested, and so on. It is hard to codify that
in a set of rules. Strict rules may actually rule out some things that would be
useful. For instance, would several proprietary implementations with
well-documented interop success qualify or not?
Jari