This reminds me: we shoul probably have some user-accessible method of
detection of the Unicode encodings, too: UTF-8 (well, this is really
guessing, at least without a BOM, "does this look like valid UTF-8 to
you"), but BOMs and UTF-16-foo, and UTF-32-foo.
Actually, UTF-8 autodetection can be pretty reliable (though I wouldn't
recommend it for applications). The chances that a string in another
encoding is completely free of malformed or overlong UTF-8 sequences are
pretty small. UTF-8 has enough unique syntactic rigor to make it quite
easily distinguishable from any other encoding.
There's a detailed recommendation on BOM handling by encoding converters
in
http://www.cl.cam.ac.uk/~mgk25/unicode.html#ucsutf
which I hope you will find helpful.
There are further recommendations for the authors of Unicode encoding
converters in
http://www.cl.cam.ac.uk/~mgk25/unicode.html#conv
especially on how to use field 5 in the Unicode database and the Unihan
database to construct correct Unicode-to-somethingelse mapping tables.
These sections summarise the intensive past discussions on these issues
on the linux-utf8 mailing list. I hope that you are already familiar
with them. If not, please read and consider the above sections
carefully. Thanks!
Markus
--
Markus G. Kuhn, Computer Laboratory, University of Cambridge, UK
Email: mkuhn at acm.org, WWW: <http://www.cl.cam.ac.uk/~mgk25/>
