[Top] [All Lists]

Re: draft-newman-i18n-collation-09.txt just posted

2006-05-10 06:28:02

Mark Davis writes:
The release of this is timely (we didn't get notified of a 07 or 08 draft), since the Unicode Technical Committee is meeting next week, and can discuss it.

Could you indicate which of the items raised in the email of 2006-02-21 from the Unicode Technical Committee have been addressed in this release (and if not accepted then why)? That would help greatly with the review. (I couldn't find any archive for discussion of draft-newman-i18n-comparator where that email could be publicly linked from, so I am appending it at the end of this message.) At a quick glance, it appears that a number of comments have been incorporated.

Lots. Some not. See below.

It is possible that some of my changes don't satisfy you. I had conflicting requests for many things. Feel free to repeat, rephrase or add arguments.


BTW, despite the subject of the message, the document is at It helps to send out a link, especially if the name (comparator vs collation) is wrong ;-)

Mea culpa. My apologies.

To:   Network Working Group
Re:   draft-newman-i18n-comparator
Date:         2006-02-21
From:         Unicode Technical Committee

The Unicode Technical Committee has reviewed the document While UTC is in favor of the goal, there are a number of problems with the document. The main problems are outlined below. Once these are addressed, then further review can continue.


      > 2.1 Definitions


The document needs to include the definitions of the technical terms used in the document, including all those that may not be familiar to implementers, such as "trichotomous" and "collation identifiers". In particular, the notion of a substring is /prima facie/ quite simple, but there are complications that require a clear definition. The text in the document does not make clear that there may be more than one match for a substring in a string, and that the matches can overlap. It says "the starting offset", for example, when there may be multiple ones.


Moreover, language sensitive matches have additional complications which need to be called out. For more information, see

Not really changed. As I recall, I added a little bit of text.


If there is a "Definitions" section, readers have a reasonable expectation that that section should contain all the required definitions. However, a number of definitions are scattered within the text. One of two approaches should be taken

   1. Move all the definitions into this section.
   2. Remove the definitions section, but clearly call out in the text
      the definitions of  each terms on its own line.

Mixing these two styles is needlessly confusing for readers.

Not changed; I'm going by what confuses reviewers.

      > 2.4 Sort Keys

The use of the term "collation canonicalization" to refer to sort keys is very misleading. ...

Changed; the text now speaks of sort keys. I'm afraid there still are instances of the old term around, I found one today.

> 3.2

This specifies that clients that support disconnected operation should not use wildcards while clients that provide collation operations only when connected to the server may use wildcards.

This restrinction has been lifted.

The EBNF syntax shown in section 3.2 says that the collation-wild must not exceed 255 characters total while the section 3.1 specifies that the collation name must not exceed 254 characters.

Brought into sync.

It seems having the same maximum possible length for both collation name and wildcard string would be desirable for actual implementations.

I picked 254, not 255, but I confess I cannot remember why.

      > 4.2.1 Equality

It needs to be made clear that the return values are not physically the strings "match", etc. but enumerated values such as /equal/ and /not_equal/.

Changed. Also other similar changes.

One extremely important point is that for a given comparator, the equality function must be synchronized with the ordering function.

I've done this and all the other equivalences/connections/implications I could see.

The term 'error' is also problematic, since what is really at issue is a question of domain. For all those strings in the domain, either 'equal' or 'not_equal' should be returned from the equality function. For any string not in the domain, 'undefined' should be returned.

Not changed. Back in February, I agreed that "error" was not ideal, but did not see "undefined" as better, and could not find a really apt term. The collations were a little too well-defined in the "undefined" cases then.

However, in -10, I think they really will be undefined outside their domain, so I'll change to using "undefined" instead of "error". (I'm removing the bits that fall back to i;octet.)

There is a typo at the 4'th line of the second paragraph of the section 4.2 saying "... For example, an collation" which should be changed to "... For example, a collation" instead.


      > 4.2.2 Substring

Prefix and suffix matching are not fully spelled out.

I think they are now.

The operations and their results must be clarified. And as noted before, it is very important to precisely define the substring operations, especially the starting offset and ending offset. It also must be clarified whether what is being asked for is the first possible matching location in the string, the last, or the nth one.

Partly changed. I didn't do the bits you ask for in the last sentence. I can add an open issue.

      > 4.3.3 Ordering

> It MUST be transitive and trichotomous.

As above, these should be defined.

I did not, since I think this document is the wrong place to define these terms.

The exposition in this section would be simpler if you also defined "reversible", whereby f(a,b) = less iff f(b,a) = greater.

The exposition changed enough as a result of other commens that I isregarded this comment.

An 'undefined' value can be allowed if, as per equality above, it means that at least one of the operands is outside of the domain. The function then imposes a total order on all strings in the domain; moreover, a wrapper can easily convert the function to a total order over all strings by putting all items outside the domain either below or above the ones in the domain -- or even excluding them,/ at its choice./

I'm doing something like this in -10. (Removing the fallback to i;octet.)

[Note: it is very important to avoid the confusion between "identical" and "equal". According to a caseless compare, "Mark" and "mark" are equal; however, the strings are not identical.]

Changed all over the place.

[Either 'ordering function' or 'comparison function' should be used consistently, not sometimes 'collations'].


      > 4.3.  Internal Canonicalization Algorithm

This section is difficult to understand.

Changed; I hope the new text is better.

      > 4.4.  Use of Lookup Tables

It is not at all clear what is meant by "customizable lookup tables".

Clarified and partly removed.

      > 4.5.  Multi-Value Attributes

This is very unclear.


This is a very important feature that needs to be spelled out in detail, and clearly reflected in the template for registration. In particular, the template should have provision for multiple attributes, with the ability to specify the acceptable operands for that attribute. (See below). The specification of the operands could be either a list of values, or a regular expression (with the former preferred). Suggested regular expression syntax would be Perl or XML Schema.

I asked Martin Dürst and you to provide a new DTD. Martin said okay, I don't remember whether you answered. I think the DTD should come before this.

      > 5.1Character Encoding

   The protocol specification has to make sure that it is clear on which
   characters (rather than just octets) the collations are used.  This
   can be done by specifying the protocol itself in terms of characters
   (e.g. in the case of a query language), by specifying a single
   character encoding for the protocol (e.g.  UTF-8 [3]), or by
   carefully describing the relevant issues of character encoding
   labeling and conversion.  In the later case, details to consider
   include how to handle unknown charsets, any charsets which are
   mandatory-to-implement, any issues with byte-order that might apply,
   and any transfer encodings which need to be supported.

If a collation is able to advertise itself as being able to handle, say, SJIS and UTF-8, then there should a required description of a protocol for indicating that and for communicating which encodings are handled, and how it handles error conditions (such as a charset outside of those it can handle. Otherwise, it is difficult to understand how this paragraph would be applied in practice.

      > 5.3

The section 5.3 specifies:

    The protocol MUST specify how comparisons behave in the absence of
    explicit collation negotiation or when a collation of "*" is
    requested. The protocol MAY specify that the default collation
    used in such circumstances is sensitive to server configuration.

and the section 3.2 specifies:

    ... If the wildcard string matches multiple collations, the server
    SHOULD select the collation with the broadest scope (preferably
    international scope), the most recent table versions and the
    greatest number of supported operations. A single wildcard
    character ("*") refers to the application protocol collation
    behavior that would occur if no explicit negotiation were used.

These appear inconsistent.


      7.5.  Example Initial Registry Summary

The sample registry would suffer a combinatorial explosion if parameters are not handled differently.

This is the DTD issue.

> 11.  Security Considerations

This is insufficient. It should at least point to the problems related in UCA and in (note that that document has been approved by the UTC and will be posted as an approved version soon.)

It now refers.


One of the real problems with the IANA character registry is that the entries are underspecified. It quite often occurs that two vendors implement the same IANA charset conversion different ways, leading to significant interoperability problems and text corruption. See, for example,

We have the real concern that this registry could lead down the same path.


> collation, it has to say so

There are places where the text should be clarified, as to whether a MUST or SHOULD is implied; this is just an example.

> "comparator" vs "collator"

Either one term or the other should be used consistently.

Collator, now.

> Unicode 3.2

Unicode 3.2 is obsolete; the the reference versions for the Collation Registry should be Unicode 5.0 and UCA 5.0, since those will be approved and published by the time the Internet Application Protocol Collation Registry has completed its review and been approved.

I'll update to the then-current versions immediately before submitting the final draft as an RFC.

Because of the use of NamePrep, it is probably the case that Unicode 3.2 also needs to be included, but strongly recommended for usage only by protocols or systems dependent on NamePrep. Note that as of UCA 4.0 and beyond, the version number of UCA is guaranteed to be identical with the version number of Unicode that it is defined for.

> Versioning

This is tricky, and should be clarified. In many instances, it is sufficient to use an unversioned collator, such as simply "UCA". In other cases, there are requirements to use a specific version, or a version of at least X. This needs to be described.

IETF documents should have only immutable references. Thus, I can reference "UCAv14", but not "UCA", because the latter moves to v15, v16 and onwards.