perl-unicode

Re: Workaround to a unicode bug needed

2010-09-06 15:11:05
Pierre Nugues schrieb am 06.09.2010 um 11:09 (+0200):
I wrote a simple tokenizer for texts containing Latin9 characters.

You probably mean "non-ASCII characters". Latin9 alias ISO-8859-15 is
the encoding. It's worth while making a distinction here.

When I run the program on a Mac Snow Leopard, with version 5.8.8 on
the text encoded in UTF-8, Perl outputs a defective UTF-8 code for
this character: <BB>

As someone said, given correct input acquisition, the output depends on
your binmode setting and your terminal. Use ":utf8" or "encoding(UTF-8)"
for a UTF-8 terminal.

### An elementary tokenizer. Save it in UTF-8

Then it should have "use utf8" indeed.

__BEGIN

while ($line = <>) { 
   $text .= $line;
}

If your data is text in UTF-8, you'll want to set your filehandle
(STDIN in this case) to UTF-8:

  binmode STDIN, 'encoding(UTF-8)'; # 'utf8' also works but less correct

$text =~ tr/a-zåàâäæçéèêëîïôöœßùûüÿA-ZÅÀÂÄÆÇÉÈÊËÎÏÔÖŒÙÛÜŸ'()\-,.?!:;/\n/cs;
  # The dash character must be quoted
$text =~ s/([,.?!:;()'\-])/\n$1\n/g;
$text =~ s/\n+/\n/g;
print $text;

I don't know your goal, but consider that \w in a regular expression
works fine to catch words:

          \,,,/
          (o o)
------oOOo-(_)-oOOo------
use strict;
use warnings;
my $fn = shift or die "Datei!"; # file in some encoding
open my $fh, '<encoding(iso-8859-15)', $fn or die "open $fn: $!";
my $txt = do { local $/; <$fh> };
close $fh;
binmode STDOUT, ':utf8'; # UTF-8 terminal
print $txt;
my @words = $txt =~ m/\w+/gms;
print $_, "\n" for @words;

-- 
Michael Ludwig