Moin Juerd,
Am 08.03.2010 um 16:15 schrieb Juerd Waalboer:
Michael Ludwig skribis 2010-03-08 15:55 (+0100):
Okay. But unless I'm completely misled, you can tell whether a
string is supposed to contain characters (<- Encode::decode) or
bytes (<- Encode::encode)
The result of decode is a character string.
The result of encode is a byte string.
Thanks for confirming.
However, apart from looking at the source code and deducing the
intentions of the programmer, there is no way to tell whether a given
string is meant as a character or byte string, simply because there is
no technical representation of this intent in the string or its
metadata.
Note that "characters" are the general case: a string is made of
characters. When every character value fits in a single byte, the string
can be used as a byte string.
And clarifying further.
This bug forces us to look at the internal encoding and flags to come to
the conclusion that it is indeed a bug. Don't mistake this as a sign
that looking at the internal encoding or flags should ever happen in
actual code. Even if you work around the bug, make sure that you don't
make anything conditional on the current formatting of the string.
Instead, coerce it to whatever you need by using utf8::downgrade or
utf8::upgrade. In your specific case, concatenation of two separate
parts is probably the most sane thing to do.
Good.
Am I mistaken in my expectation that while "\xa0" should be
a byte, "\x{a0}" and "\x{00a0}" should be characters?
Yes. These three escapes are supposed to be exactly the same. They
create a U+00A0 character, which happens to be perfectly usable as the
A0 byte when used as such, in a string that doesn't contain any
character greater than U+00FF.
Okay. Let me try to see if I have understood correctly. Without the utf8
pragma in scope, "so\xa0ein\xa0Käse" with a-Umlaut stored as a sequence
of two bytes in my source code will be stored internally as a sequence
of 12 integers. With the utf8 pragma in scope, only 11 integers.
I know I shouldn't care about the internals, but sometimes grokking the
internals is helpful as an aide-mémoire, because it puts things into
perspective that otherwise seem more arbitrary.
--
Michael.Ludwig (#) XING.com
