On Oct 23, 2004, at 01:04, Bjoern Hoehrmann wrote:
C12a in Unicode 4.0.1 notes
[...]
For example, in UTF-8 every code unit of the form 110xxxx must be
followed by a code unit of the form 10xxxxxx. A sequence such as
110xxxxx 0xxxxxxx is illformed and must never be generated. When
faced with this ill-formed code unit sequence while transforming or
interpreting text, a conformant process must treat the first code
unit
110xxxxx as an illegally terminated code unit sequence--for example,
by signaling an error, filtering the code unit out, or representing
the code unit with a marker such as U+FFFD
[...]
[snip]
Okay, you win. You have convinced me that Encode::utf8 should behave
the same as Encode::XS (UCM-base encodings). And the patch to make
that way is deceptively simple, as follow;
===================================================================
RCS file: Encode.xs,v
retrieving revision 2.0
diff -u -r2.0 Encode.xs
--- Encode.xs 2004/05/16 20:55:15 2.0
+++ Encode.xs 2004/10/22 18:00:29
@@ -297,7 +297,7 @@
U8 skip = UTF8SKIP(s);
if ((s + skip) > e) {
/* Partial character - done */
- break;
+ goto decode_utf8_fallback;
}
else if (is_utf8_char(s)) {
/* Whole char is good */
@@ -313,6 +313,7 @@
/* Invalid start byte */
}
/* If we get here there is something wrong with alleged UTF-8 */
+ decode_utf8_fallback:
if (check & ENCODE_DIE_ON_ERR){
Perl_croak(aTHX_ ERR_DECODE_NOMAP, "utf8", (UV)*s);
XSRETURN(0);
===================================================================
The most decisive comment of yours is this:
holds true and I expect that
my $x = "Bj\xF6rn"; # as well as "Bj\xF6r" and "Bj\xF6"
decode("utf-8", $x, Encode::FB_CROAK);
croaks.
Which apparently did not. Thank you for being so persitent on this
problem. I'd be honor to add your name to AUTHORS file for this.
I will $Encode::VERSION++ as soon as I am done w/ the test suites and
Tel's patch. This time I will be careful not to screw up
(maint|bread)perl so give me some time before the update is ready (but
I won't keep you waiting for too long since 5.8.6 deadline is soon).
Your statement about \xF6\x80\x80\x80 is interesting, Encode::is_utf8
is
documented as
[...]
is_utf8(STRING [, CHECK])
[INTERNAL] Tests whether the UTF-8 flag is turned on in the STRING.
If CHECK is true, also checks the data in STRING for being
well-formed UTF-8. Returns true if successful, false otherwise.
[...]
And D36 in Unicode 4.0.1 is very clear that
[...]
As a consequence of the well-formedness conditions specified in Table
3-6, the following byte values are disallowed in UTF-8: C0–C1, F5–FF.
[...]
That's because perl's notion of Unicode is broader than that of
unicode.org. So far Unicode.org's mapping only spans from U+0000 to
U+1fFFFF, While that of perl is U+ffffFFFF or even U+ffffFFFFffffFFFF
(in other words, MAX_UINT). See Camel 3 on details.
And I think we can leave this :)
Dan the Encode Maintainer
