On Thu, Apr 29, 2004 at 10:42:18PM -0400, Lincoln A. Baxter wrote:
On Thu, 2004-04-29 at 11:16, Tim Bunce wrote:
Am I right in thinking that perl's internal utf8 representation
represents surrogates as a single (4 byte) code point and not as
two separate code points?
This is the form that Oracle call AL32UTF8.
What would be the effect of setting SvUTF8_on(sv) on a valid utf8
byte string that used surrogates? Would there be problems?
(For example, a string returned from Oracle when using the UTF8
character set instead of the newer AL32UTF8 one.)
I think it makes no difference. (at least I could no find one), except
for the internal storage. Several of the tests I wrote print a sql
DUMP(nch), and you can see the difference in the internal store in those
prints. The strings come back to the client, the way they were put in.
I have tested this with 4 databases
dbcharset/ncharset
--------- --------
us7ascii/utf8
us7ascii/all6utf16
utf8 /utf8
utf8 /al16utf16
All tests produce the same results with all databases using both .UTF8
and .AL32UTF8 in NLS_LANG.
Were you using characters that require surrogates in UTF16?
If not then you'd wouldn't see a difference between .UTF8 and .AL32UTF8.
Here's a relevant quote from the Oracle 9.2 docs at
http://www.dbis.informatik.uni-goettingen.de/Teaching/oracle-doc/server.920/a96529/ch6.htm#1005295
"You can use UTF8 and AL32UTF8 by setting NLS_LANG for OCI client
applications. If you do not need supplementary characters, then it
does not matter whether you choose UTF8 or AL32UTF8. However, if
your OCI applications might handle supplementary characters, then
you need to make a decision. Because UTF8 can require up to three
bytes for each character, one supplementary character is represented
in two code points, totalling six bytes. In AL32UTF8, one supplementary
character is represented in one code point, totalling four bytes."
So the key question is... can we just do SvUTF8_on(sv) on either
of these kinds of Oracle UTF8 encodings? Seems like the answer is
yes, from what Jarkko says, because they are both valid UTF8.
We just need to document the issue.
Tim.
p.s. If we do opt for defaulting NLS_NCHAR (effectively) if NLS_LANG
and NLS_NCHAR are not defined then we should use AL32UTF8 if possible.
