On Thu, Aug 15, 2002 at 05:28:43PM -0400, David Gray wrote:
I'm having a bit of a problem getting Unicode pattern
matching to do what I would like it to.
I guess my question wasn't entirely clear. I'm reading in the attatched
file and trying to split it on "\n\n".
When I'm looping over the file,
I've (sort of) made it work by doing:
# strip BOM and trailing nulls and carriage returns
s/^..// if $. == 1 and s/\0//g;
s/[\0\r]//g;
The two-byte BOM has me thinking it's probably UTF-16. Is there an easy
way to tell what encoding a file uses?
Not that I know of, but all the 0 bytes make me think it is.
But I'm sure there must be a more elegant way to do this.
Honestly, I'm not even sure where to start. Any ideas?
I find that this:
perl5.6.1 -we 'undef $/; $_=<STDIN>; $_ = pack "U*", unpack "v*", $_; substr
($_, 0, 1) = ""; print $_' </tmp/unicode.txt
gives me this:
fdn "grp1",55,"","",0
fdn "grp2",55,"","",0
fdn "grp3",55,"","",0
fdn "grp4",55,"","",0
fdn "grp5",55,"","",0
fdn "TEMP",55,"","",0
The substr takes out the byte order mark.
I guess a better conversion script would read the first two characters, and
if they look like a byte order mark in UTF-16 chose whether to use v or n
in the unpack based on the endianness.
You will get more sane regexp behaviour if you use 5.8.0 rather than 5.6.1
In 5.6.1 being in the scope of a "use utf8;" will make your regexps properly
unicode, even if they don't contain obvious Unicode features.
(Otherwise matches involving . and similar metachars cause the regexp to think
in ASCII, and unicode scalars are treated as a series of bytes.
5.8.0 fixes this problem - regexps "just work" there. Modulo unknown bugs)
Nicholas Clark
--
Even better than the real thing: http://nms-cgi.sourceforge.net/
