On 30 Jun 2004, at 04:11, Edward Batutis wrote:
Malformed UTF-8 character (unexpected non-continuation byte 0x73,
immediately after start byte 0xe9) in substitution iterator at
/usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi/HTML/
Entities.pm
line 435, <DATA> line 1.
Segmentation fault
I think this is an internal 'utf-8 flag' problem. I'm not a Perl
internals
expert, but there seems to be some funny goings on internally where
Latin-1
strings are sometimes stored as Latin-1 and sometimes as UTF-8. The
magic
UTF-8 flag has to match the internal representation of the string. In
this
case, the UTF-8 flag is set, but the internal representation is not
UTF-8
but Latin-1. That causes the low-level string parser to barf.
It might be possible to find a workaround by converting explicitly to
UTF-8
and maybe manually setting the flag. I'm not sure that'd help though.
See
the Encode docs for more info on how you would do this.
I tried doing explicit conversions, but didn't get very far. I did come
up with a screwy work around, however.
In my original mail the offending line was:
<title>The Modern R&eacute;sum&eacute;</title>
Now this is a bit off, because is RSS, therefore utf8, but its got
encoded latin1 entities (é) in there, with the & further encoded
for xml safety.
After xml parsing the & are resolved. Then I used decode_entities
to resolve the é, then encoded again - and thats where it
crashed. (I had to encode again because _most_ of the rss feeds I'm
dealing with don't start with encoded entities).
Anyway, if I drop the initial decode, and do s/&([A-Za-z])+;/&$1;/g
after the encoding I solve the double encoding problem without causing
a segfault.
So it looks like:
parse xml
$title = $item->title;
# decode_entities( $title ) # removed this line
encode_entities( $title )
$title =~ s/&([A-Za-z])+;/&$1;/g # added this one
Richard
=Ed Batutis
