Jarkko Hietaniemi wrote:
On Tue, Dec 11, 2001 at 11:34:09AM -0800, Brian Stell wrote:
Jalal,
Kindly reply via the mailing list so others can see the discussion.
That way others can benefit and/or help.
BOM is the Byte Order Mark used in Unicode to indicate an
important detail about the Unicode data stream.
Perhaps the Perl people can describe how to inhibit the BOM?
I don't think it's Perl putting the BOM in there.
I opened up Notepad in Win2000, wrote "foobar", and saved the file
as "ANSI", "UTF-8", "Unicode", and "Unicode big endian". Then in UNIX
with this
perl -e 'print "$ARGV[0]: "; print unpack "H*", <>; print "\n"' file.name
I get
foo.ansi: feff0066006f006f006200610072000d000a
foo.utf8: efbbbf666f6f6261720d0a
foo.unic: fffe66006f006f006200610072000d000a
foo.unib: feff0066006f006f006200610072000d000a
(copied by hand, so typos possible) which looks like little-endian
UTF-16, UTF-8, big-endian UTF-16, and (again) little-endian UTF-16
to me. For example the "Unicode" is first the BOM, then the 0x66
aka "f", then two 0x6f:s, aka "o", then 0x62, aka "b", and so on.
No Perl was involved in creating these files, but the BOMs are there
(the UTF-8 0xEF 0xBB 0xBF is the BOMin disguise).
Moreover, if the browser claims to do Unicode, it should recognize the
BOM, too, and ignore it in display (but of course use it to figure out
the right endianness).
The BOM is valid as the *first* character. I'm not sure what the
spec says about subsequent chars.
How did the browsers handle the foo.* files?
Of course you will may need to manually set the encoding to get
proper results since these do not have a charset tag. I do believe
that the Netscape 6.2 universal autodetector should detect it
automatically (when turned on).
--
Brian Stell
