The IETF mandates the use of UTF-8 for text [RFC2277] as part of
internationalisation. When writing an RFC, this raises a number of issues.
A) Character set. UTF-8 implicitly specifies the use of Unicode/IS10646 which
contains 97,000 - and rising - characters. Some (proposed) standards limit
themselves to 0000..007F, which is not at all international, others to
0000-00FF, essentially Latin-1, which suits many Western languages but is not
truly international. Is 97,000 really appropriate or should there be a
defined
subset?
Short answer: No.
Longer answer: Subsetting makes sense in some places, such as when defining
canonicalization schemes like our various *prep profiles. You pretty much have
to limit yourself to what's currently defined for these and Unicode is, for
better or worse, open-ended.
However, limiting everything to a subset of Unicode simply because "97,000
characters are too many" is a terrible idea. This has been tried - take a look
at the various string types for Unicode in ASN.1 or the text body part types in
X.400 - and the result has been nothing but a mess.
B) Code point. Many standards are defined in ABNF [RFC4234] which allows code
points to be specified as, eg, %b00010011 %d13 or %x0D none of which are
terribly Unicode-like (U+000D). The result is standards that use one notation
in the ABNF and a different one in the body of the document; should ABNF allow
something closer to Unicode (as XML has done with �D;)?
ABNF is charset-independent, mapping onto non-negative integers, not
characters. Nothing prevents a specification from saying that a given ABNF
grammar specifies a series of Unicode characters represented in UTF-8 and using
%xFEFF or whatever in the grammar itself.
C) Length. Text is often variable in length so the length must be determined.
This may be implicit from the underlying protocol or explicit as in a TLV.
The
latter is troublesome if the protocol passes through an application gateway
which wants to normalise the encoding so as to improve security and wants to
convert UTF to its shortest form with corresponding length changes
The various length issues and tradeoffs that exist in different Character
Encoding Schemes for Unicode are well known, have been extensivey debated, and
are well understood. There are inherent tensions between the various formats
(preserve ASCII vs. equal length for all characters vs. wasting space)
and this makes any choice a compromise.
(Unicode
lacks a no-op, a meaningless octet, one that could be added or removed without
causing any change to the meaning of the text).
NBSP is used for this purpose.
Other protocols use a terminating sequence. NUL is widely used in *ix; some
protocols specify that NUL must terminate the text, some specify that it must
not, one at least specifies that embedded NUL means that text after a NUL must
not be displayed (interesting for security). Since UTF-8 encompasses so much,
there is no natural terminating sequence.
This simply isn't true. NUL is present in Unicode and is commonly used as a
terminator.
D) Transparency. An issue linked to C), protocols may have reserved
characters,
used to parse the data, which must not then appear in text. Some protocols
prohibit these characters (or at least the single octet encoding of them),
others have a transfer syntax, such as base64, quoted-printable, %xx or an
escape character ( " \ %). We could do with a standard syntax.
I disagree. Different environments have very different constraints, and in many
cases these constraints interact with the underlying character data in ways
that force the use of different escaping conventions. For example, the
differences between quoted-printable the content-transfer-encoding and the Q
encoding of encoded-words are NOT gratuitous.
E) Accessibility. The character encoding is specified in UTF-8 [RFC3629]
which
is readily accessible (of course:-) but to use it properly needs reference to
IS10646, which is not. I would like to check the correct name of eg
hyphen-minus (Hyphen-minus, Hyphen-Minus, ???) and in the absence of IS10646
am
unable to do so.
The entire Unicode character database is readily available online:
http://www.unicode.org/ucd/
A quick check shows that 0x002D is written as HYPHEN-MINUS in the database and
in the code charts.
I didn't need (and never have needed) a copy of ISO 10646 to find out stuff
like this. I do find a copy of the printed Unicode book to be useful but not
required (and I have versions 1 through 3), but it is readily available, albeit
not free.
The reason many of our standards documents refer to ISO 10646 is that at one
time there was concern that Unicode wasn't sufficiently stable, and it was felt
that reference to the ISO document would offer some protection against
capricious change. I think in retrospect this concern has been shown to be
unwarranted, and all things being equal I would prefer to see references to the
more readily available Unicode materials. (Given the wide deployment of Unicode
now there is effectively no chance of a major change along the lines of the
Hangul reshuffle between V1 and V2.)
Overall, my perception is that we have the political statement - UTF-8 will be
used - but have not yet worked out all the engineering ramifications.
Well, we have a lot more than a political statement - a huge amount of
engineering work has been done to make Unicode workable in IETF protocols and
elsewhere. Does more work need to be done? Of course it does - these tasks are
by their very nature pretty much unending. But all of the points you have
raised here are either nonissues, settled issues, or engineering compromises
where there is no "right" answer.
Ned
_______________________________________________
Ietf mailing list
Ietf(_at_)ietf(_dot_)org
https://www1.ietf.org/mailman/listinfo/ietf