In regard to source data for perl encodings:
I have a lot of experience with encodings and encoding converters on a lot of
platforms, and compatibility between these converters can turn into a huge
mess. Because Perl can be used to create essentially permanent data
repositories, compatibility across platforms is a good thing.
This is why using one single source for code page conversions is important and
why using platform-specific conversions is a bad thing.
One could pick a set of encoding files from a lot of places, but it seems to me
that using the ICU data files has some key advantages: 1) there's someone to
complain to when they are wrong - and I'm sure there will be bugs found in them
and 2) ICU's data files are being used by real software so using them isn't
quite so bleeding edge plus 3) they are under source control and have
versioning so you can say 'this data was created with ICU data files version
N.M' - at least someday you could say that somehow.
If there are licensing issues, I think they can be resolved. If you try to
contact the ICU team about this and you don't get anywhere, please let me know
and I'll try to help.
I believe the best thing long term would be to use ICU for all conversions.
Given this, it makes sense to use the ICU data files in the short run so you
can hope for least controlled incompatibility.
And I would like to point out that although ICU's converter data file is
largish, it doesn't need to be. It isn't hard to trim it back or to even use
separate table files. Also, ICU doesn't load the whole table data file into
memory at once or some silly thing like that. It tries to be efficient.
So, to summarize, I think there is a need for built-in conversion tables at
this point in time and I believe they should be derived from ICU UCM files.
Also, I believe that only a small set of single-byte encodings should be built
in. Multibyte and ISO-2022 related encodings - even only those used just on the
Internet - are amazingly nasty and best left to something like ICU. Even ICU
doesn't get everything right, but there's a decent process to fix things that
are wrong.
In regard to the built-in converter engine, it needs these key features
(assuming single byte and Western/Central European languages plus Cyrillic
only):
1) convert a single byte code point to Unicode.
2) convert several 'close cousins' from Unicode to a single character code. In
other words, it needs to convert Unicode N to x and Unicode M to x, etc. as
separate characters. The ICU UCM files have these alternate mappings in them.
3) allow Perl internals (at least) to specify what to do when you can't
translate a character from Unicode to the single byte encoding. This is
important because you might introduce interesting bugs/security holes when, for
example, a question mark gets splotched into your regular expression or, if you
strip untranslatables, when the match string becomes empty. This seems like a
potentially big mess to me, but hopefully it really isn't.
4) a non feature for this language group is converting Unicode combining
sequences (or simply multiple Unicode characters) to single characters in the
code page (and vice versa). This is required for some encodings, but not in
this language group. (There are cases where this would be nice, but this isn't
a critical feature.)
Thanks and regards,
=Ed
------------------------------------------------------------
--== Sent via Deja.com ==--
http://www.deja.com/
