g'day,
Mark Adam wrote:
It almost sounds like we want to reward the WGs which complete their work
while producing the _least_ amount of documentation. If we assume that a
document is "good" and "complete" then the most concise representation
should be the easiest to work with.
Ok... So I'm being a little idealistic, but this is different that just
saying "Me too" to the "We ain't makin' widgets" responses. Optimally we
should judge the work of a WG based on how well its output is accepted by
the world at large, but that's a little late in the process.
I like this a lot, and the fact that you're judging success only after
the work in done within the IETF context based upon metrics outside your
direct control doesn't discount the value of this measurement for me. On
the contrary, I think it is an important reason it has value.
Taking a step back for a moment, see why are we interested in measuring
something? You measure to see how effective you are at reaching your
goals. So what are the goals of any IETF activity? Presumably it's *not*
to produce RFCs, or to exchange ideas, or even to consume pastry. I
respectfully submit for your flaying and filleting that the core goal of
the IETF is to promote interactive communications through the
development of open protocols.
Now, the wording on that last sentence may suck, but if you can agree
that this is in any way within the gravity well that catpures the IETF's
goals, then isn't the real measure of the success of any IETF activity
the degree in which new protocols are adopted and old ones enhanced?
Sure, you can only determine this after the IETF work is done, but
that's because the core question is "do people use it?"
The implications for this seem clear enough. It seems to imply that the
amount of traffic per protocol the activity goes on to generate is a
reasonable milestone for any IETF activity. This doesn't mean the POISED
list (or heck, even the IETF general list ;-) should be shut down (and
putting aside the question of the increase in SMTP traffic they generate
:-) The point is that we should recognize that such activities are
clearly overhead, and we should all be doing our best to minimize such
overhead whenever we can.
So, if people agree that traffic measurements have value as a metric,
then presumably the first derivative of traffic volume over time is also
a reasonable indicator of the future takeup rate for new protocols.
Measuring some of the other things being discussed here (RFC counts,
engineer-hours spent in meetings, messages to a mailing list, count of
pastries consumed) would all seem to me to be measuring overhead
activities, not core to the organization. This is not a bad thing to
understand, but would not seem to be the most important metric in our
arsenal.
If any of this makes sense, then the good news is that this sort of
thing is eminently measureable. The bad news for some activities or
protocols is that despite the best efforts of the participants, the
track record of takeup is not there. Still, we shouldn't be shy about
wanting to know this. We can then try to investigate what the numbers
are trying to tell us.
One more thought and then my turn on the soapbox will be over. One of
the things I like about focusing on factors which are outside the IETF's
control, such as traffic measurements, is that they tell us a lot about
how individuals are "voting with their packets" on the IETF's success in
fulfilling their core mandate. This keeps us honest. Instead of allowing
us to conclude that we're doing fine and those pesky users are letting
us down by not deploying our favorite new toy, it puts the
responsibility for convincing users to adopt our work clearly on our own
shoulders, which is where I think it belongs. Just as any good business
person knows the the difference between a good technology and a good
product, we should acknowlege the difference between a good technology
and a good solution. If people don't deploy it, we are the ones who
failed. We didn't build it in such a way that it was a better
alternative for the user and it wasn't used. Anything else is noise, not
signal.
As a simple test, if we were to find that the percentage of traffic on
the net using IETF developed/endorsed protocols turns out to be falling,
it would imply that the organization's influence is waning, which would
be something we might want to investigate. If it is rising, it would
suggest that the model is perhaps more successful than some people seem
to think right now. It would move us away from talking about how *hard*
something is, and towards determining how *successful* it is, and let us
focus on improving ourselves in ways that we can determine. Of course,
this implies a faith in the user that is not shared by everyone on this
list....
- peterd
mark--------------------------
At 3/28/02 16:01, Bill Strahm wrote:
I am reminded that early in my career I was in a company that was driven by
the
KLOC metric. They had determined that the product would have 150ish KLOC
in it
and so had every programmer report the number of KLOC they had contributed
that week.
One week I was looking through the code I had inherited and realized that I
had two
copies of a set of utilities that did the same code. I spent a day or two
removing
one set, and porting that half of code to use the other set of utilities
(Basically
I had inherited two developers code). Well my KLOC for the week was
somewhere in
the -10 range, and it was a month before I started going positive again. My
reviews
sucked, but it was the right thing to do.
Becareful what you measure, because that is the behaviour you will get
Bill
--
-----------------------------------------------------------------------
Peter Deutsch peterd(_at_)gydig(_dot_)com
Gydig Software
"This, my friend, is a pint."
"It comes in pints?!? I'm getting one!!"
- Lord of the Rings
----------------------------------------------------------------------