-----Original Message-----
From: Curtis Villamizar [mailto:curtis(_at_)ipv6(_dot_)occnc(_dot_)com]
Sent: Tuesday, January 21, 2014 2:23 PM
To: Black, David
Cc: ops-dir(_at_)ietf(_dot_)org; ietf(_at_)ietf(_dot_)org;
draft-ietf-mpls-multipath-
use(_dot_)all(_at_)tools(_dot_)ietf(_dot_)org
Subject: Re: OPS-Dir review of draft-ietf-mpls-multipath-use-03
In message
<8D3D17ACE214DC429325B2B98F3AE712026F047819(_at_)MX15A(_dot_)corp(_dot_)emc(_dot_)com>
"Black, David" writes:
I have reviewed this document as part of the operations directorate's
ongoing effort to review all IETF documents being processed by the
IESG. These comments were written primarily for the benefit of the
operations area directors. Document editors and WG chairs should treat
these comments just like any other last call comments.
Summary:
This draft is on the right track but has open issues, described in the
review.
I apologize for this review showing up a couple of days after the end of
IETF Last Call.
This draft discusses considerations for what are effectively MPLS-in-MPLS
tunnels for multipathing - an LSP carries another LSP, with the inner
(client) LSPs being distributed across multiple outer (server) LSPs that
use different paths. This draft is motivated by considerations specific
to MPLS-TP that complicate multipathing when one of the two MPLS instances
is MPLS-TP.
Skipping to the discussion quite a ways down, there seems to be some
confusion about what is being proposed here. While MPLS-in-MPLS is
being used, no new form of MPLS-in-MPLS is being proposed.
There is an underlying assumption in the draft that the reader
understands that MPLS-in-MPLS exists today, most commonly but not
exclusively in the form of RFC 4206 hierarchy. RFC 4206 provides a
means to advertise forward adjacencies (FA) where MPLS-in-MPLS at some
level of Packet Switch Capable (PSC) hierarchy. It is also possible
to make use of the management plane, using CLI or NETCONF or the MPLS
cross connect MIB to program a multiple label MPLS push operation.
If a very experienced reviewer could miss that, then the introduction
could be improved by better stating assumptions. It is also the case
that many participants in MPLS, particularly those primarily
interested in MPLS-TP, began participation long after RFC 4206 was
written and would benefit by better stated assumptions in the intro.
I suggest that the following paragraphs be added to the introduction.
MPLS LSPs today are able to function as a server layer and carry
client MPLS LSPs. When control plane signaling is used, forwarding
adjacency (FA) advertisements are used to inform the set of LSR of
Packet Switching Capable (PSC) LSP within the MPLS topology
[RFC4206]. Client MPLS LSP at a higher layer (lower PSC number)
may signal their intention to use PSC LSP as hops in the RSVP-TE
Explicit Route Object (ERO). LSR with no explicit support for RFC
4206 see the PSC LSP as ordinary links and therefore use them.
An MPLS LSP that has been set up using RSVP-TE appears to its
ingress LSR as a viable IP next hop to a distant LSR. If LDP is
used and bidirectional RSVP-TE LSP connectivity is available, then
LDP signaling can be set up among the RSVP-TE LSP endpoints and LDP
can make use of the RSVP-TE LSP as an LDP hop. This is another
form of existing MPLS-in-MPLS use.
MPLS LSPs may also make use of hierarchy that is configured through
the management plane rather than signaled using RSVP-TE.
These existing forms of MPLS-in-MPLS may traverse multipath hops
such as Ethernet LAG [IEEE802.1AX] or MPLS Link Bundling [RFC4201].
MPLS-TP brings with it a new set of requirements not considered in
past deployments of the various forms of MPLS-in-MPLS where
multipath was in use. This document merely discusses use of
existing forwarding and protocol mechanisms that can support the
case where either the client layer LSPs or the server layer LSPs
are MPLS-TP and where multipath is used.
If there are no objections I will add these paragraphs.
Now on to the specific comments.
I consider the following to be open issues that need attention (each
letter is used to tag the issue in the text that follows - [B] occurs
multiple times, as there are multiple aspect to that issue):
[A] Summary text in Introduction appears to be wrong.
[B] Section 3.2 needs a summary.
[C] Entropy label "sandwich" seems peculiar and at least needs an
explanation.
[D] Discussion of multipath impacts on MPLS OAM should be added to Section
4.
--- Comments on draft text
[A] Section 1 (Introduction) 2nd paragraph:
RFC 5654 requirement 33 requires the capability to carry a client
MPLS-TP or MPLS layer over a server MPLS-TP or MPLS layer [RFC5654].
This is possible in all cases with one exception. When an MPLS LSP
exceeds the capacity of any single component link it MAY be carried
by a network using multipath techniques, but MAY NOT be carried by a
single MPLS-TP LSP due to the inherent MPLS-TP capacity limitation
imposed by MPLS-TP Operations, Administration, and Maintenance (OAM)
fate sharing constraints and MPLS-TP LM OAM packet ordering
constraints (see Section 3.1).
Section 3.1 concerns carrying MPLS-TP over MPLS, whereas the above
exception appears to concern the reverse, and Section 4 on carrying MPLS
over MPLS-TP allows for MPLS LSPs that exceed component link capacity.
Was the exception intended to be for an MPLS-TP LSP that exceeds the
capacity of any single component link?
There is no problem today with MPLS LSP carried by MPLS-TP LSP except
the existing case where MPLS LSP capacity is too big. For example, a
500 Gb/s LSP is possible today carried over N x 100 Gb/s or N x 10
Gb/s multipath links but could not be carried over a single MPLS-TP
LSP which would be limited to a single 100 Gb/s or 10 Gb/s link. So
the exception is where the MPLS LSP is too big to fit in any possible
single MPLS-TP LSP.
To answer your question directly: the exception is a very large MPLS
LSP can't be carried by a single MPLS-TP LSP. The solution is to use
multiple MPLS-TP LSP. The MPLS-TP LSR need not be not "aware" that
this is occurring.
Also, "MAY NOT" is not an RFC 2119 term (for good reason - it's equivalent
to "MAY OR MAY NOT"). I think "cannot" was intended here, but "MUST NOT"
is another possibility.
Good point. I suggest we replace the MAY NOT with SHOULD NOT. Many
people would insist that it should be MUST NOT, but in reality
networks SHOULD NOT be congested but sometimes they are.
[B] Section 3.2 contains a lot of useful details, but it could really use
a summary at the end on the options and recommendations for meeting the
four requirements (this summary could be a new Section 3.3). It's a bit
difficult to discern what's going on, as I had to read this section more
than once in order to discern the following:
OK. A Summary paragraph can be added. See below. Perhaps conclusion
section can be added. See far below.
[C] In Section 3.2, use of an MPLS entropy label to meet requirement MP#1
requires that it be "below the MPLS-TP label" whereas, to meet requirement
MP#2, the MPLS entropy label needs to be "just above the MPLS-TP LSP label".
This combination appears to suggest "sandwiching" the MPLS-TP label between
two entropy labels that effectively carry the same information (albeit,
applied and removed at different places in the network) - was that intended?
This is important, as there seems to be no reasonable alternative
to the entropy label for meeting requirement MP#2, and the draft's
alternative to the entropy label for meeting requirement MP#1 is "some form
of configuration" - the entropy label would seem to be preferred to that
based on my reading of this draft and the comment in RFC 5706 section 2.2
that "Anything that can be configured can be misconfigured."
The "MPLS MPLS-TP ELI EL" sandwich is suggested in the following
paragraph.
Alternately, the need for requirement MP#1 can be eliminated if
every MPLS-TP LSP created by an MPLS-TP ingress makes use of an
Entropy Label Indicator (ELI) and Entropy Label (EL) below the
MPLS-TP label [RFC6790]. This would require that all MPLS-TP LSR
in a deployment support Entropy Label, which may render it
impractical in many deployments.
Please not the "Alternately" at the beginning and "impractical in many
deployments" at the end.
The other sandwich "MPLS ELI EL MPLS-TP" is described two paragraphs
above that but assumes knowledge of RFC 6790. I think the following
change within this paragraph would clear things up:
OLD
MPLS server layer LSP ingress LSR an Entropy Label Indicator (ELI)
and Entropy Label (EL) are added to the label stack [RFC6790]. For
those client LSP that are MPLS-TP LSP, a single EL value must be
chosen. For those client LSP that are MPLS LSP, per packet entropy
below the top label must, for practical reasons, be used to
determine the entropy label value.
NEW
MPLS server layer LSP ingress LSR an Entropy Label Indicator (ELI)
and Entropy Label (EL) are added to the label stack by the midpoint
LSR for the client MPLS-TP LSP, at the ingress of the MPLS LSP
[RFC6790]. For those client LSP that are MPLS-TP LSP, a single EL
value must be chosen. For those client LSP that are MPLS LSP, per
packet entropy below the top label must, for practical reasons, be
used to determine the entropy label value. The resulting label
stack contains the server MPLS LSP label, ELI, EL and the client
LSP label.
The above rewording is simply more explicit about the label stack
order and which LSR puts the ELI and EL in the label stack.
[B] A summary of requirements for implementation and deployment at the end
of Section 3.2 would have made this much more obvious. That should be
provided in addition to addressing the above question about duplicate use
of the MPLS entropy label.
The following summary can be added to the end of Section 3.2:
Two methods of adding an Entropy Label are described above. The
MPLS-TP ingress must have a means to determine which links can
support MPLS-TP in selecting a path (MP#4). Administrative
attributes can satisfy that requirement. If the MPLS-TP LSR is
capable of adding ELI/EL to the label stack, this method is
preferred. However access and edge is the least likely to support
RFC 6790 in the near term. For portions of the topology where an
MPLS-TP is carried within a server layer MPLS LSP the ingress of
the server layer MPLS LSP can add ELI/EL using a fixed EL value per
client LSP, except those known not to require MPLS-TP treatment.
There are numerous ways to determine which client LSP are MPLS-TP
LSP and which are not. While this determination is out of scope
and will vary among deployments, configuration or the presence of
specific attribute affinities in RSVP-TE signaling are among the
likely means to do so.
If there are no objections I will add the above paragraph.
--- Relevant Q&A based on RFC 5706 Appendix A
I'm omitting management concerns, as both MPLS and MPLS-TP have extensive
management infrastructure, and this draft's notion of stacking LSPs so
that one instance of MPLS (or MPLS-TP) can be a server to (i.e., carry)
another is not novel to this draft.
A.1.1. Has deployment been discussed?
[B] Yes, although Section 3.2 is weak on deployment requirements (some of
which are implementation requirements) and could use a summary.
The use of RFC 4206 is assumed throughout the discussion of signaling
in section 3.2. This assumption is made more clear by suggested
additions to the introduction.
A.1.2. Has installation and initial setup been discussed?
No, although because this draft reuses existing functionality in different
configurations, this really reduces to the A.1.9 configuration question
below. In practice, quite a bit of operator knowledge and insight is
required to get initial setup right, e.g., as traffic engineering LSPs to
avoid congestion problems (needed to meet requirement MP#3) requires
significant knowledge of network structure and expected traffic flows.
This should be obvious to anyone who works w/MPLS, but it's probably
worth stating for completeness.
FA advertisements defined in RFC 4206 make it possible for ingress to
signal viable paths. Either the use of Ethernet Link Aggregation or
MPLS Link bundling using the all-ones component link (RFC 4201) allows
the total capacity of a multipath to be advertised. Meeting the
requirement to balance traffic over the component links in a multipath
(MP#3) is then a local matter and requires no further signaling
extensions.
A.1.6. Have suggestions for verifying correct operation been discussed?
Yes and no. Section 3.2 discusses OAM, and contains this important
requirement for OAM traffic:
An Entropy Label must be used to insure that all of the
MPLS-TP payload and OAM traffic are carried on the same component,
except during very infrequent transitions due to load balancing.
[D] OTOH, there is no discussion of OAM traffic for the MPLS LSP in
Section 4; that should be added.
There is no impact on MPLS OAM traffic. There are three cases to
consider.
1. Where MPLS is carried over a single MPLS-TP all traffic flows on
one link. MPLS OAM is unaffected and need not use multipath
support in LSP Ping.
2. Where MPLS-TP is carried over MPLS, all traffic for that MPLS-TP
LSP is carried over one link thanks to the fixed EL value.
MPLS-TP OAM is unaffected.
3. Where MPLS is carried over MPLS (an existing case) or over
multiple MPLS-TP, the multipath support in LSP Ping (as
imperfect as it may already be) is used and LSP Ping (RFC 4379)
operation is unaffected.
If needed something can be added to state the above (maybe with less
informal wording). What is obvious to some people may not be obvious
to everyone. Briefly stating the almost obvious is usually not a bad
thing. Please tell me if a small subsection on "Imopact on MPLS and
MPLS-TP OAM" is needed.
Note that the limitations of LSP Ping are currently being discussed in
the MPLS WG in the context of extensions to LSP Ping to better support
Entropy Label. A few limitations not related to EL may get fixed in
the process.
A.1.9. Is configuration discussed?
[B] Not much. Configuration is required to meet requirements MP#3 and MP#4,
and is one of the options for MP#1. A summary of what has to be configured
as part of the summary at the end of Section 3.2 would be a good idea.
Perhaps a worked example appendix could be added but I don't think
this is common and may not be necessary with some of the
clarifications suggested so far.
A.2.* [skipped, see above]
A.3 Documentation
There's quite a bit of operational material contained in this draft; a
separate section on operations and management considerations would not
be appropriate.
There is a useful implementation status section which appears to imply
that MPLS-TP over MPLS is not currently deployable due to absence of
entropy label support, and because deployed MPLS equipment does not
generally support multipathing. Nonetheless, data center experience
indicates significant value and widespread use of multipathing based
on link aggregation and ECMP; corresponding benefits can be expected
for use of multipathing in the Internet beyond data centers.
If you would like I can add a conclusion section as Section 5 with
these paragraphs:
MPLS equipment deployed in the core currently supports multipath.
For large service providers, core LSR must support some form of
multipath to be deployable. Deployed MPLS access and edge
equipment is often oblivious to the use of multipath in the core.
It is expected that at least first generation MPLS-TP equipment
will be oblivious to the use of multipath in the core. This first
generation MPLS-TP equipment is deployable in a core using
multipath, with no adverse impact to RSVP-TE signaling, if the edge
equipment can support administrative attributes (RFC 3209) and the
core equipment can support ELI/EL and put a per-LSP fixed EL value
on any LSP that indicates a particular attribute affinity or can
identify client MPLS-TP LSP through some other means.
There are no issues carrying MPLS over MPLS-TP, except when the
MPLS LSP is too big to be carried by a single MPLS-TP LSP. Most
MPLS core equipment and some edge equipment can configure an MPLS
Link Bundle [RFC4201] over multiple component links where the
component links are themselves MPLS LSP. This existing capability
can be used to carry large MPLS LSP and overcome the limited
capacity of any single server layer MPLS-TP LSP.
If this conclusion section is added, then the three bullets above
regarding impact on OAM can be put there. The three bullets can be
added as follows:
MPLS OAM and MPLS-TP OAM are unaffected in the following cases
proposed in this document:
1. Where MPLS is carried over a single MPLS-TP all traffic flows
on one link. MPLS OAM is unaffected and need not use
multipath support in LSP Ping.
2. Where MPLS-TP is carried over MPLS, all traffic for that
MPLS-TP LSP is carried over one link thanks to the fixed EL
value. MPLS-TP OAM is unaffected.
3. Where MPLS is carried over MPLS (an existing case) or over
multiple MPLS-TP, the multipath support in LSP Ping is used
and LSP Ping (RFC 4379) operation is unaffected.
I think adding the above as a conclusion section as Section 5 would
improve clarity and therefore is worth doing.
--- Editorial comments/nits:
- Section 3.1, after RFC 5960 quotes:
[RFC5960] paragraph 3 requires that packets within a specific ordered
Insert "Section 3.1.1.", before "paragraph 3" and likewise for "paragraph 6"
later in the same paragraph in the draft.
GenArt review caught this. Done.
- Section 3.2, requirement MP#3:
MP#3 It SHOULD be possible to insure that MPLS-TP LSPs will not be
"insure" -> "ensure" or "assure". Surely this draft does not envision
operators taking out insurance policies on MPLS TP LSP behavior ;-).
Good catch. I think "ensure" is the right word.
- Section 3.2, last paragraph on p.8:
An MPLS-TP LSP may not traverse multipath links on the path where
MPLS-TP forwarding requirements cannot be met.
"may not" is meaningless - I believe "MUST NOT" was intended here.
I suggested SHOULD NOT above.
- Section 4, middle of 3rd para:
Editorial suggestions for clarity:
OLD
For those LSP that are larger than component link
capacity, their capacity are not integer multiples of convenient
capacity increments such as 10Gb/s.
NEW
For those LSPs that are larger than a component link
capacity, the LSP capacities need not be (and often are not)
integer multiples of convenient capacity increments such as 10Gb/s.
Good suggestion. GenArt review suggested adding "generally" making
the statement read "their capacity are generally not integer
multiples". Your suggested rewording may be more clear.
Also "10Gb/s" should be "10 Gb/s".
In particular, "are not" seems wrong, as the "integer multiples"
case is possible, so I changed "are not" to "need not be (and often
are not)" in the suggested new text.
Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA 01748
+1 (508) 293-7953 FAX: +1 (508) 293-7786
david(_dot_)black(_at_)emc(_dot_)com Mobile: +1 (978) 394-7754
----------------------------------------------------
Thanks for the review and for pointing out lack of clarity due to
underlying assumptions about the use of RFC 4206 (and perhaps use of
RFC 4201 as well for large MPLS LSP over MPLS-TP).
Sometimes it is too easy to forget that not all readers are going to
be as steeped in MPLS "as deployed today" details as a relatively
small number of long time MPLS participants are.
Adding a few paragraphs to the introduction and a few concluding
paragraphs may significantly improve clarity for people who have not
been following MPLS closely for a decade or more without requiring
them to read the references and figure out usage details themselves.
Please let me know how you feel about the suggested additions to the
document.
Curtis