|
Re: Review of: draft-resnick-on-consensus-05
2013-10-10 17:49:07
Finally back to this original review.
On 10/6/13 7:03 PM, Dave Crocker wrote:
In terms of philosophy and desirable practice, the draft
discusses an extremely appealing model and generally explains its
nature and practice well. However the draft tends to confuse what is
(or has been) with what should be. This includes confusing the IETF's
historical sense of what 'rough consensus' means, with what the draft
suggests it mean. As a consequence, it sometimes merely explains what
is preferred, rather than the problems with what is common practice.
(Sometimes. At other times, it nicely makes its case.) Also it
sometimes does not lay out the context for its assertions.
In effect, the draft is offering fundamental enhancements to
existing IETF practice, rather than merely clarifying the ways things
are "supposed to be".
I think this is a fair criticism and I will do a grand editing pass to
try to tease the two apart: Where the draft is actually describing real
historical practices vs. where it is describing a theory that does not
comport with our practices. I know of several places beyond what you've
mentioned that make the same conflation, and I think it's well worth
making it clear. There are some instances where "what is (or has been)"
is actually mushy: That is, people haven't really thought about it in
depth. In those cases, I've definitely done a bit of saying what I think
folks would mean if they were actually asked about it. (See more on this
point in the discussion of the intro.) But with all of these, I think I
should be more explicit about which is which.
By way of properly setting context, the draft should cite RFC 1603/3934
You mean 2418. But yes, I think you're right.
Working Group Guidelines and Procedures. Its section 3.3 has text
that describes long-standing community views on the meaning of rough
consensus, which don't quite align with what the draft puts forward:
In general, the dominant view of the
working group shall prevail. (However, it must be noted that
"dominance" is not to be determined on the basis of volume or
persistence, but rather a more general sense of agreement.) Consensus
can be determined by a show of hands, humming, or any other means on
which the WG agrees (by rough consensus, of course). Note that 51%
of the working group does not qualify as "rough consensus" and 99% is
better than rough. It is up to the Chair to determine if rough
consensus has been reached.
Yup. I think this document has to cite that statement in 2418, and it
has to disagree with it, at least to the extent of claiming it is an
over-simplification. I had intended to do so and simply forgot. That
51%/99% statement actually leads down the path that the document wants
to discourage.
The disparity between that document's text and what this draft
describes should not be a problem for what the draft is recommending,
but merely requires changing the explanatory tone of the draft and how
it makes its case.
[...]
The changes being suggested to the draft are substantial, but
will not affect the essential points for which the draft is lobbying.
I agree, though I think the changes are actually pretty straightforward.
Maybe substantial in number of actual edits, but not in essential content.
Abstract
a "majority rule-> a "simple majority rule"
Majorities come in different forms or degrees and the fact that 'rough
consensus' is often taken to mean 67% or 75%, as a rule of thumb can
make this confusing.
From what you've written, your basic point seems to be that 51% isn't
enough; it's worth making that explicit.
Setting aside the possible confusion in British usage, I don't think you
understood my intention here. And since you and Christopher and Jaap
ended up in the same place, I'm happy to say that it's my fault for not
being clear enough.
I intended that the abstract simply summarize the document's *purpose*,
not its conclusions. I was using it like the abstracts that appear on
scientific papers. Of course, the abstracts on our technical
specifications usually give away the punch line, which is what we're
used to seeing. I'll have a go at an edit, at least to clarify what I'm
doing.
For example, "voting" can be done with models that give quite a bit of
power to minorities. So, you need an explicit, direct, early
statement that asserts the nature and importance of "rough consensus"
factoring in minority views.
This is an abstract, not a place to put early statements. I think the
kind of statement you're looking for goes in the intro, which I
hopefully address below.
1. Introduction
[...]
That is, our credo is that we don't let a single individual make the
As the draft phrases it, that isn't really true. We let individuals
make decisions all the time. What I believe the aphorism is saying --
and what reflects our community belief -- is: We do not assign to any
one individual the authority to /dictate/ a decision. ("make" is too
soft a word for this point, IMO.)
Fair enough.
decisions, nor do we let the majority dictate decisions, nor do we
I believe the model being pursued in this draft -- of the nature and
essential importance of properly dealing with minority views -- is
hugely powerful and appropriate. But I believe it has never been a
prevalent view in the community, including back when the aphorism was
seeking to summarize the community view.
Yes, I think this all goes to your earlier point of cleanly
distinguishing between historical view and new theory.
The language I remember from the 90s primarily distinguished "strongly
dominant majority" from 51% or 67% or 75%. That is, it was about
imprecise nature of the required clear and strong majority, rather
than the moderating role of the minority. Very occasionally -- but
only occasionally -- we would use language that was along the lines of
"strongly dominant view, with no vigorous and coherent opposition".
When I started having conversations about the draft with folks,
especially people who've been around quite a while, I was struck by the
number of times the discussion started with, "No, that's not what we
meant", but ended in, "Oh, yes, in those extreme cases, that's exactly
what we meant." That is, I think the "51% is not enough, but 99% is more
than enough" view was certainly in the front of people's minds, but when
probed about the implications of that, almost everyone came to the same
conclusion as the document: 51/99 is often a good rule of thumb if one
wanted to guess about the consensus based on numbers of people, but it's
still reasonable to call certain outcomes (like the ones described in
the document) "rough consensus" even if the views of a huge majority
might not win the day.
But it is certainly fair to say that in the 90s, few of us would have
independently articulated the idea that a 90% majority might not be
rough consensus.
That latter part probably should be seen as a crude version of what
the draft is seeking to discuss, but I believe it was never
consistently voiced or understood.
In other words, I think the role of minority views that the draft is
putting forward is not a mere clarification and refinement to the
existing IETF model, but rather is a significant enhancement. So its
premise about the potential importance of minority views does have
some historical hooks in the IETF philosophy, but not as much as the
draft seems to assert.
Yep. I hope I can edit to make that clearer.
allow decisions to be made in a vacuum without practical experience.
Again, this statement goes beyond actual practice. The language here
prohibits decisions without experience and of course, the real IETF
has always indulged in some decision-making based on theory that
lacked practice. We /prefer/ experience. We /value/ the input it
provides. But we are perfectly capable of making decision in a vacuum.
Remember that I'm saying what the "credo" says, not how tightly we
adhere to it. How well we stick to it is talked about later. I think we
do take as a principle that we want to base decisions on running code.
We certainly don't always do so.
Instead, decisions are made by (more or less) consent of all
participants, and the actual products of engineering trump
theoretical designs.
"Consent of all"? Hell no. Not even close, as your following text
already note.
Again, here it is stating what the credo says, not the specifics of what
we do (or even mean to do) in practice.
[Compare: Person A: "The US Constitution says, 'Congress shall make no
law...abridging the freedom of speech.' Our credo is that the government
will not limit speech." Person B: "But that's not true. The government
limits all sorts of speech." Person A: "Never said it didn't. I said
what the credo was. I will talk at length in a moment about how the
government does and does not adhere to the credo, and how some of that
is fine and still inline with our overall principles, and how some of it
is problematic, and how they can do things that are less problematic."]
This is the Introduction. It's setting up the bigger picture. It's not
the place to say, "The details are complicated"; of course they are.
Our ideal is full consensus, but we don't
require it: Full consensus would allow a single intransigent person
Full consensus would allow
->
Requiring full consensus gives each person a veto, by allowing
Right. I'm inclined to skip the "veto" language, but I will definitely edit.
who simply keeps saying "No!" to stop the process cold. We only
require rough consensus: If the chair of a working group determines
Again, this seems to conflate two points. Both are extremely
important. Both need to be discussed. But not intermixed.
One is that we do not require everyone to agree, as long as there is a
strongly dominant group that do agree.
The other is that all concerns must be diligently considered, but do
not have to be thoroughly resolved. 'Diligently considered' is an
imposition on the entire group but it is mandatory. 'Not thoroughly
resolved' is acceptable when the strongly dominant group agree that it
is acceptable.
All true, but that discussion is not appropriate to the Introduction.
The distinction between the two points is discussed at length later.
that a technical issue brought forward by an objector has been truly
considered by the working group, and the working group has made an
informed decision that the objection has been answered or is not
enough of a technical problem to prevent moving forward, the chair
can declare that there is rough consensus to go forward, the
objection notwithstanding. To reinforce that we do not vote, we have
"To reinforce" switches from discussing the handling of minority
concerns about voting. At a minimum, break the paragraph.
Done.
also adopted the tradition of "humming": When (for example) the chair
of the working group wants to get a "sense of the room", instead of a
show of hands, the chair asks for each side to hum for or against a
question.
The draft has historically drawn a strong distinction between hands
and hums. It's an interesting distinction, and well might be useful,
but I believe it has not been a universally-shared distinction in the
community. It essentially relies on a theoretical analysis of
differences in psychological effects, but there's no objective basis
for validating the one really is superior to the other.
And the draft says as much. See section 4, paragraphs 3-5. This is
discussed at length. I don't see what's missing from here.
More and more
frequently, people walk away from working groups, thinking that
"consensus" has created a document with horrible compromises to
satisfy everyone's pet peeve instead of doing "the right thing".
None of these things are indicators of a rough consensus process
being used, and are likely due to some basic misperceptions.
Is the issue their perception, or the fact that horrible compromises
are actually being done? From the text, I can't tell what point is
intended.
Here, it's only about the perception. I talk about the real problems
with compromises at the end of section 2.
it is not intended to dictate -> it does not dictate
Will fix.
2. Lack of disagreement is more important than agreement
The substance of this section has an important focus and makes useful
comments.
However the title could be taken to cover such things as failure to
gain affirmative support.
I don't intend any of my section titles to be complete and precise. If
folks really don't like that, I can go about changing them, but I'd
really rather not.
... Consensus is not when everyone is
par break at start of sentence.
Sure.
And possibly move this and the remaining sentences higher up, since
they voice the essential point of the section. In other words, try to
start with the premise so that folks are oriented, and then explain it.
I know this is apropos of a conversation we're having offline, but I do
believe that people can get through the example of the problem before we
tell them what the solution is.
Here's a suggestion for some re-casting to this separate paragraph, to
help the reader bit, with the draft's logic and meaning:
Engineering always involves a set of tradeoffs. The process of
gaining agreement about tradeoffs is the essence of a good rough
consensus effort, and requires distinguishing the group debate and
compromise needed to _come to_ consensus, the chair's process of
_determining_ (or assessing) consensus, and the final point of the
group's _having_ consensus on it choices.
Did you intend for that to replace something or just to insert it? I
think it works well inserted, but if you meant something to be deleted,
please clarify.
However, there is another sense of "compromise" that involves
compromising between people, not engineering principles.
This seems to argue that political compromise is never acceptable.
That's not realistic. Folks often can "live with" unresolved
concerns. The meta-issue is distinguishing disagreements about key
points versus disagreement about minor ones. Political compromise
about a flat vs. hierarchical namespace is a rather different kettle
of importance than between specifying a syntactic separator of comma
vs. semicolon. (No matter how much prettier I might think commas are,
and superior for good visual presentation...)
So, the "live with" case is mentioned above, but I think it needs
clarification in the previous paragraph. But I *do* want to argue that
political compromise *is* never acceptable. It's one thing for me to
conclude that I can live with semicolons over commas even though my
preference is for commas (i.e., that it is a reasonable, though not
preferred, engineering choice); that's just fine. It's quite different
for me to conclude that the rest of these people are idiots for not
choosing commas, but I'm going to put up with the semicolons because I
want to make people happy so they'll do a hierarchical namespace, or
because I've simply given up on trying to make this a good document. If
it's a serious issue, giving up or trading is bad. If it's an "I can
live with it" issue, then you're simply agreeing that the choice is
acceptable.
And I'm not saying that the bad sort of compromise won't ever happen (we
are humans, after all), or that there are no borderline cases (Is this
me "living with it" or capitulating?), but I do want to say that
participants have to decide whether they're making a reasonable
engineering compromise or they are capitulating (or horse trading), and
not do it if it's the latter.
And all that said, I need to clarify this.
Note that this portrays rough consensus as an "exception".
I don't understand this first sentence. How is it an exception? Even
with the following text, I don't see the basis for claiming that it
might/would be seen as exceptional.
Maybe I need a different word. I didn't mean "exceptional" in the sense
of "extraordinary" or "atypical". I simply meant it in the "exception
handling" sense: We work toward full agreement, but if there's still
disagreement, we undertake an alternative path. I'm open to suggestions
for text.
Now, a conclusion of having only rough consensus relies heavily on
the good judgement of the consensus caller. The group must truly
consider and weigh an issue before the objection can be dismissed as
being "in the rough". The chair of the working group in one of these
In spite of being appealing to use here, I think that the phrase "in
the rough" is actually distracting
As I mentioned earlier, I think it is worth defining as the golf term,
since it is used quite frequently among IETFers nowadays.
cases is going to have to decide that not only has the working group
taken the objection seriously, but that it has fully examined the
ramifications of not making a change to accommodate it, and that the
outcome does constitute a failure to meet the technical requirements
of the work. In order to do this, the chair will need to have a good
idea of the purpose and architecture of the work being done and use
their own technical judgement to make sure that the solution meets
those requirements. What can't happen is that the chair bases their
decision solely on hearing a large number of voices simply saying,
"The objection isn't valid." That would simply be to take a vote. A
valid justification needs to me made.
This begs the question of who determines that a justification is
valid. I'll suggest that it mostly should be the working group, with
the chair as fall-back or safety valve, but not default.
Remember, this is precisely the case where the working group claims to
have considered the objection and found it to be wanting, but the
objector still insists that the objection is valid. The rest of the
working group can make the case that it has properly considered the
objection, but it can't make the final call on whether it has; only the
chair can do that.
By the way, the draft seems to think that it's always the chair who
leads discussion and debate; it often/mostly isn't. Rather it's
typically the authors/editors, with a chair serving to moderate impasses.
During my large edit, I'll have a look at this. My intention wasn't to
say that the chair *leads* discussion. Indeed, beyond authors/editors,
quite often it is someone else in the group who takes up the mantle of
analyzing the discussion and proposing paths forward. But the chair in
the end is responsible for making the final call as to the consensus of
the group.
A good outcome is for the
objector to understand the decision taken and accept the outcome,
even though their particular issue is not being accommodated in the
final product. Remember, if the objector feels that the issue is so
par break at Remember.
Sure.
So, why do we hum?
This section reads nicely and sounds quite good and very possibly
might be something that we /should/ do. However it has little to do
with common IETF practice.
Yep. This goes back to your main critique. So really the above should
say, "So why should we hum?", or something to that effect. Wilco.
Even better is that I think the draft has buried it's more important
point, namely that the /form of the question/ asked when seeking a hum
is the really hard part of the effort, and the most important part.
Some forms invite constructive discussion and some close it off. Some
are fair and balanced (...) and some are highly biasing.
I'll try to expand on that.
But couldn't the above could have been done with a show of hands
This paragraph goes back to the hum-vs-hand methodology issue. It's
important, but it's out of place here, since this section is about the
/role/ of a hum, not its method. Discuss methodology separately,
possibly up where it is first introduced.
I disagree with you on this one. The previous discussion was in the
Introduction, which says, "Something's screwy here. What's up with
that?" Here, I am trying to say, "Here's what we ought to be doing. Hums
are good. Hands work too, but have their weaknesses." I think it works
together well.
"the chair wants to choose the path that gets to the least objections
fastest" sounds like an extremely appealing expedient, by virtue of
being almost mechanical, but numbers often don't matter. A single
objection that observes (and substantiates) that a choice will not or
cannot be used in the real world will outweigh a much larger number of
trivial objections.
Perhaps merely qualifying the statement a bit will suffice: the chair
wants to choose the path that gets to the least number of significant
objections fastest". Hmmm.
Yeah, this gets talked two paragraphs previously, but I think it's fair
to reiterate here. I think the better fix is to add to the below sentence:
There is always a chance
that this could be misleading, because some people are more willing
to hum loudly than others (just by dint of personality), but keep in
mind that taking the hum is to figure out how to start the
conversation.
Add after the parenthetical: "or that one of the quieter hums actually
turn outs to be a showstopper that makes the original choice impossible,"
Of course, a chair could get the temperature of the room by doing a
show of hands too, and knowing who specifically feels one way or
another can help a good chair guide the subsequent conversation.
And possibly place the minority under group pressure.
Sometimes a hum can do exactly the same.
Sometimes, they simply mean they are making a call _of_ the
consensus; that is, they are declaring the consensus that has, in
their view, been reached when the discussion has reached an end.
That's a fine thing and what chairs are supposed to do: They are
"calling" the consensus. Sometimes, when a chair says that they are
We often hear chairs say that they are making a "consensus call".
Might be worth tossing in the word "confirm" or "confirming existing
consensus" here, to leave no doubt about what "calling" means.
Yeah, or use the word "stating" or "expressing". I'll work on that.
So in a large working group with over 100 active participants and
broad agreement to go forward with a particular protocol, if a few
participants say, "This protocol is going to cause congestion on the
network, and it has no mechanism to back off when congestion occurs;
we object to going forward without such a mechanism in place", and
the objection is met with silence on the mailing list, there is no
The problem here is who is the authority to determine that the
objection needs to hold sway. It's easy to lay this on the chair, but
that makes the mechanisms fragile; it's too easy for one person to get
this sort of thing wrong.
The case we're talking about here is an objection being met with
silence. In that case, the presumption (i.e., not the conclusion) is,
"Real objection; needs to be dealt with." The only way that the chair
can "get this sort of thing wrong" is to at that point silently decide
that it's not an objection worth addressing. The chair can say to the
group, "I've seen this objection, but I don't think it's valid. Can
someone confirm my view that it's not valid, or can the objector or
someone else please explain why it is?" But this is only about whether a
completely dropped-on-the-floor objection is OK. It's not.
Did I miss your point?
7. Five people for and one hundred people against might still be rough
consensus
...
Now the worst case scenario happens. The objector, still unhappy
that their preferred solution was not chosen, recruits one hundred
people, maybe a few who were silent participants in the working group
already, but mostly people who work at the same company as the
objector who never participated before. The objector gets them all
to post a message to the list saying, "I believe we should go with
the new elegant algorithm in section Z instead of the current one.
It is more elegant, and works better on our hardware." The chair
sees these dozens of messages coming in and posts a query to each of
them: "We discussed this on the list, and we seemed to have consensus
that, given the inherent risk of a new algorithm, and the widespread
deployment of this current one, it's better to stick with the current
one. Do you have further information that indicates something
different?" And in reply the chair gets utter silence. These
posters to the list (say some of whom were from the company sales and
marketing department) thought that they were simply voting and have
no answer to give. At that point, it is within bounds for the chair
to say, "We have objections, but the objections have been
sufficiently answered, and the objectors seem uninterested in
participating in the discussion. Albeit rough in the extreme, there
is rough consensus to go with the current solution."
There's an interesting methodological point buried quite deep in this
example, and probably worth surfacing sooner and more directly:
Besides being careful about how the question for a consensus
check is phrased, it is sometimes advisable to seek to discover what
the responses mean.
In the example here, the question is whether those responding were
sufficiently well-informed to be able to justify their position.
I'll take an edit pass. Yes, that is exactly what I was trying to get at
(among other things).
There is no doubt that this is the degenerate case and a clear
indication of something pathological. But this is precisely what
rough consensus is designed to guard against: vote stuffing. In the
Sorry, no. It's not 'designed' for that at all, at least not in
typical practice in the IETF. It probably /should/ be, and the draft
does a pretty good job of explaining how it /can/ be, but there is
nothing in the "design" of the IETF's consensus process that has a
history of mitigating against the problem the draft discusses here,
and the problem isn't mentioned in formal IETF documents.
Yes, fair criticism. "But this is precisely what rough consensus is
ideally suited to guard against:"
Excellent comments. Thanks.
On to the next set....
pr
--
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478
|
|