perl-unicode

RE: Variation In Decoding Between Encode and XML::LibXML

2010-06-18 05:45:41
So it may be valid UTF-8, but why does it come out looking like crap? That 
is, "LaurinaviÃ?Â?ius"? I suppose there's an > argument that "LaurinaviÄ?ius" 
is correct and valid, if ugly. Maybe?

I am unsure if this is the explanation you are looking for but here goes:

I think the original data contained the character \x{010d}. In utf-8, that 
means that it should be represented as the bytes \x{c4} and \x{8d}. If those 
bytes are not marked as in fact being a two-byte utf-8 encoding of a single 
character, or if an application reading the data mistakenly thinks it is not 
encoded (both common errors), somewhere along the transmission an application 
may decide that it needs to re-encode the characters in utf-8. 

So the original character \x{010d} is represented by the bytes \x{c4} and 
\x{8d}, an application thinks those are in fact characters and encodes them 
again as \x{c3} + \x{84} and \x{c2} + \x{8d}, respectively. Which I believe is 
your broken data.

I think the error comes from Perl's handling of utf-8 data and that this 
handling has changed in subtle ways all the way since Perl 5.6. We have 
supported utf-8 in our applications since Perl 5.6 and have experienced this 
repeatedly. Any major upgrade of Perl or indeed the much needed upgrade of 
DBD::ODBC Martin Evans provided have given us a lot of work trying to sort out 
these troubles.

I wonder if your code would work fine in Perl 5.8? We are "only" at 5.10(.1) 
but the upgrade from 5.8 to 5.10 also gave us some utf-8 trouble. If it works 
fine in Perl 5.8 maybe the error is in an assumption somewhere in XML::LibXML?


Best regards
Henning Michael Møller Just

<Prev in Thread] Current Thread [Next in Thread>