Dear Karl,
It finally worked and I would like to thank you for your patience and your
help. I could apply it to any text read from the standard input and avoid the
CO options with these three lines in the beginning of the program.
use utf8;
binmode(STDOUT, ":encoding(UTF-8)");
binmode(STDIN, ":encoding(UTF-8)");
The program must read the standard input explicitly, which means that
perl token_perl.pl text.txt
will not work. It has to be
perl token_perl.pl < text.txt
instead.
Also, it is also pretty curious that the program worked fine in its original
form, for all the characters with the exception of the quote » and that the
problem disappeared when I removed the û character from the tr// list. Now
everything works fine without this hack.
Thank you!!!!
Pierre
