ietf
[Top] [All Lists]

Re: [aqm] Last Call: <draft-ietf-aqm-fq-codel-05.txt> (FlowQueue-Codel) to Experimental RFC

2016-03-22 11:49:52
Toke,

Sorry for not yet sending the follow-up. Straight after that email, I got roped into becoming a makeshift ambulance driver and then ... long story...

Thanks for taking my comments constructively, as intended. Responses embedded.

On 18/03/16 12:47, Toke Høiland-Jørgensen wrote:
Hi Bob

Thank you for your timely and constructive comments. Please see the
inline responses below.

My main concern is with applicability. In particular, the sentence in
section 7 on Deployment Status: "We believe it to be a safe default
and encourage people running Linux to turn it on: ...". and a similar
sentiment repeated in the conclusions. "and we believe it to be safe
to turn on by default, as has already happened in a number of Linux
distributions."

Can one of the authors explain why a solution with the limitations in
section 6 can still be described as "safe"?
"We believe it to be a safe default" means that we have not seen any of
the theoretical limitations we have documented in section 6 be a concern
*in practice* in any of the extensive number of deployments FQ-CoDel has
seen already. And that the benefits of turning on FQ-CoDel are
sufficient that nudging people in that direction is a good idea.
This is perhaps because "we" (ie the people looking) tend to have significantly more bandwidth than the majority of Internet users (those in the developing world). When you have less bandwidth, long-running flows last longer, so they tend to overlap more. Given bloat problems are only seen intermittently in the first place [Hohlfield14], the average person isn't going to see these limitations very often. But if you are a homeworker using a VPN (for instance), you will be dogged by these problems all the time.

So the main problem here is with the assumption that the test has to be "whether we observe these limitations in practice".

Few people observed problems with NATs at the time they were introduced (otherwise they wouldn't have sold successfully). So those arguing against them tended to be ignored by mainstream comms engineers. But then the "theoretical" limitations started to bite. And we ended up having to make do with a subset of the potential of the Internet. Those sounding the warning bells could see the potential of the Internet, and they could see how NATs would close that off. Those ignoring the warning bells believed they were right to only be concerned with the here and now.

My concern is about precluding future desirable developments in application behaviour. It will be rare to observe such cases by random inspection, they may not appear while using existing applications on existing high speed links. But, they will occur very frequently in scenarios prone to them. That's often the nature of side-effects.

My concern is particularly about fq technology in the network precluding improvements in the quality of regular best efforts service that we can expect through changes in applications and transports alone.

When I was arguing against FQ_CoDel (back in 2013 at the latency workshop - you were there too), numerous people were saying that FQ_CoDel is much more subtle than regular FQ. At which point I quietened down, because I trusted enough of those people. However, in the recent tests with HAS (criticised at length elsewhere), one thing that can be said with certainty was that FQ_CoDel just becomes a regular fq scheduler when you have two or more long-running flows that can always keep their queues from emptying. Whatever instantaneous rate the application tries to run at, FQ overrides it and runs at 1/N of the capacity. That is not good for a video coming off a camera at a variable information rate. FQ skims off all the peaks, so the VBR codec adapts down to the worst-case peak rate, not the worst-case average rate.

Indeed, these sentences seem rather Orwellian.
I can assure you that we are not attempting to exert "draconian control
by propaganda, surveillance, misinformation, denial of truth, and
manipulation of the past" (quoting
https://en.wikipedia.org/wiki/Orwellian here). But thank you for
implying it :)
Well, stating the limitations in the draft, then denying their truth in the conclusions by using the word safe to describe them is classic Orwellian Newspeak <https://en.wikipedia.org/wiki/Newspeak#To_control_thought>.

Would it not be correct instead to say that FQ_CoDel has been made the
default in a number of Linux distributions despite not being safe in
some circumstances?
At the time it was made the default in OpenWrt (several years ago now,
if memory serves me right), there was not a whole lot of real-world
deployment experience, due to the chicken-and-egg problem of not wanting
to change the default before we have gathered more experience. However,
today the situation is quite different, thanks in part to the boldness
of the OpenWrt devs. So no, I do not believe that to be the case any
longer.
The experience that led me to understand this problem was when a bunch of colleagues tried to set up a start-up (a few years ago now) to sell a range of "equitable quality" video codecs (ie constant quality variable bit-rate instead of constant bit-rate variable quality). Then, the first ISP they tried to sell to had WFQ in its Broadband remote access servers. Even tho this was between users, not flows, when video was the dominant traffic, this overrode the benefits of their cool codecs (which would have delivered twice as many videos with the same quality over the same capacity.

Now, by your test, you will never see the limitations these videos suffered. Because they never got developed. Because the developers gave up. You can think of FQ_CoDel as nice well-meaning people (the Linux community) creating a new middlebox problem.


2. Default?

If a draft saying "We believe it to be a safe default..." is published as an
RFC, it means "The IETF/IESG/etc believes..."
Only one solution can be default, so if the IETF says that FQ_CoDel is a safe
default, and no other AQM RFC makes any claim to being a safe default (which
they do not at the moment), it could be read as the IETF recommending FQ_CoDel
for default status and, by implication, other AQMs (like PIE, say) are not
recommended for default status.
This is certainly not my reading. This is an experimental RFC saying "we
believe it to be safe as a default" not a standards track RFC saying
"this should be the default". This is an important difference; we are
not mandating anything, but rather expressing our honest opinion on
the applicability of FQ-CoDel as a default, should anyone wish to make
it one in their domain.

As far as I know, unlike the listed FQ_CoDel limitations, no
limitations of PIE have been identified. I don't think anyone is
claiming that the performance of FQ_CoDel is awesomely better than
PIE. May be a bit better, may be a bit worse, depending on
circumstances, and depending on which you value most out of low
queuing delay, high utilization, or low loss.
Well, for CoDel and PIE that is certainly true. But FQ-CoDel in many
cases reduces latency under load by an order of magnitude compared to
both of them, while improving throughput.
OK, I have seen such figures, and it makes sense that FQ will give single RTT flows v low latency.

My concern is that of course the IESG will want to sign off an RFC with this cool performance, given they read that the limitations are not important. Whereas I believe the limitations have been downplayed.

So, if the authors want the IETF to recommend a default AQM on the
basis of safety (and I agree safety is the most important factor when
choosing a default), the most likely candidate would be PIE, wouldn't
it? FQ_CoDel has unintended side-effects, which implies it is not a
good candidate for default; it should only be configured deliberately
by those who can live with the side-effects.
I'm not sure it would be possible for the AQM group to agree on a
recommendation for a default. But I suppose it might be a good
bikeshedding exercise. And as noted above, this is not what we intend to
do in this case.
If we don't want the IETF (or the AQM WG) to make this call, we should make it clear that we are not making this call.

My concern is that, years down the line, when the context has been lost, these sentences could be interpreted as making this call. For comparison, consider how we have been trying to understand what RFC2309 (the RED manifesto) intended to say.


3. A Detail

I also have a concern about the way the limitations are written
(typically, each limitation is stated, followed by a arm-waving
qualification attempting to create an impression that there is not
really a limitation). To keep the thread clean, I'll send that in a
follow-up email.
It is certainly not our intention to "create an impression that there is
not really a limitation". Rather, we are trying to suggest ways in which
each limitation can be mitigated by people who are concerned about it,
but still want to realise the benefits of deploying FQ-CoDel. Sure, some
of those proposals are not exactly at the "running code" stage, but
dismissing them as arm-waving is hardly fair.

I'll add, as I noted initially, that many of the limitations we have
noted are of a theoretical nature (in the sense that we are not aware of
any deployments where they have caused issue in practice). This does not
make it any less important to document them, of course, and we have been
grateful for the feedback from the working group that the section grew
out of (you yourself were among the people providing this feedback, I
believe). However, this also means that it is difficult to do more than
point out each issue. We can't quantify them, for instance.

If you have concrete suggestions for language that would make things
clearer, do tell (though I suppose that's exactly what you'll do in your
follow-up mail). :)
See the next email (like I promised before).

Cheers


Bob


[Hohlfeld14] Hohlfeld, O., Pujol, E., Ciucu, F., Feldmann, A. & Barford, P., "A QoE Perspective on Sizing Network Buffers," In: Proc. Internet Measurement Conf (IMC'14) pp.333-346 ACM (November 2014)


-Toke

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/