I am daydreaming that I am a caravan member, driving a herd of
disobedient camels on the never-ending desert to an oasis called 5.8.0
when I released new Encode and PerlIO::encoding. You can get one as
follows.
Whole:
Encode
http://www.dan.co.jp/~dankogai/Encode-1.50.tar.gz
and CPAN
PerlIO::encoding
http://www.dan.co.jp/~dankogai/PerlIO-encoding-0.02.tar.gz
Diff
Encode
http://www.dan.co.jp/~dankogai/current-1.50.diff.gz
PerlIO::encoding
[ none ]
Diff is pretty big (> 3000 lines) so you should get a whole thing
instead.
The biggest and the foremost change is the fallback API which is greatly
enhanced. NI-XS request of
On Friday, April 19, 2002, at 05:01 , Nick Ing-Simmons wrote:
check == 11 - silent fail with $string updated (What Tk wants)
is implemented as FB_QUIET. see below;
====
Handling Malformed Data
THE CHECK argument is used as follows. When you omit it,
it is identical to CHECK = 0.
CHECK = Encode::FB_DEFAULT ( == 0)
If CHECK is 0, (en|de)code will put substitution char-
acter in place of the malformed character. for UCM-
based encodings, <subchar> will be used. For Unicode,
\xFFFD is used. If the data is supposed to be UTF-8,
an optional lexical warning (category utf8) is given.
CHECK = Encode::DIE_ON_ERROR (== 1)
If CHECK is 1, methods will die immediately with an
error message. so when CHECK is set, you should trap
the fatal error with eval{} unless you really want to
let it die on error.
CHECK = Encode::FB_QUIET
If CHECK is set to Encode::FB_QUIET, (en|de)code will
immediately return proccessed part on error, with data
passed via argument overwritten with unproccessed
part. This is handy when have to repeatedly call
because the source data is chopped in the middle for
some reasons, such as fixed-width buffer. Here is a
sample code that just does this.
my $data = '';
while(defined(read $fh, $buffer, 256)){
# buffer may end in partial character so we append
$data .= $buffer;
$utf8 .= decode($encoding, $data, ENCODE::FB_QUIET);
# $data now contains unprocessed partial character
}
CHECK = Encode::FB_WARN
This is the same as above, except it warns on error.
Handy when you are debugging the mode above.
perlqq mode (CHECK = Encode::FB_PERLQQ)
For encodings that are implemented by Encode::XS,
CHECK == Encode::FB_PERLQQ turns (en|de)code into
"perlqq" fallback mode.
When you decode, '\xXX' will be placed where XX is the
hex representation of the octet that could not be
decoded to utf8. And when you encode, '\x{xxxx}' will
be placed where xxxx is the Unicode ID of the charac-
ter that cannot be found in the character repartoire
of the encoding.
The bitmask
These modes are actually set via bitmask. here is how
FB_XX are laid out. for FB_XX you can import via "use
Encode qw(:fallbacks)" for generic bitmask constants,
you can import via
"use Encode qw(:fallback_all)".
FB_DEFAULT FB_CROAK FB_QUIET FB_WARN
FB_PERLQQ
DIE_ON_ERR 0x0001 X
WARN_ON_ER 0x0002 X
RETURN_ON_ERR 0x0004 X X
LEAVE_SRC 0x0008
PERLQQ 0x0100 X
Unemplemented fallback schemes
In future you will be able to use a code reference to a
callback function for the value of CHECK but its API is
still undecided.
====
Since PerlIO::encoding was uncapable of using this new feature, I have
updated PerlIO::encoding as well; Instead of pushing &PL_sv_yes to
stack, now struct PerlIOEncode has one more member, chk, that is
initialized with Encode::FB_QUIET.
typedef struct {
PerlIOBuf base; /* PerlIOBuf stuff */
SV *bufsv; /* buffer seen by layers above */
SV *dataSV; /* data we have read from layer below */
SV *enc; /* the encoding object */
SV *chk; /* CHECK in Encode methods */
} PerlIOEncode;
Encode now checks the version of PerlIO::encoding and refuse to use an
obsolete version. see t/perlio.t on details.
That way PerlIO::encode has no trouble should Encode changes the value
of FB_QUIET.
As for the partial character problem, I have found it is nearly
impossible for escape-based encodings to support fixed-length buffer.
That is well-documented in Encode::PerlIO, a new pod that is added.
As a workaround a new method perlio_ok() is added. You can check if the
encoding in question works well with PerlIO. I know that's not a
perfect solution but good enough for 5.8.0. The ultimate solution to
this problem is for PerlIO to implement line buffer FOR BOTH
DIRECTIONS. But that's out and beyond my realm as yet....
And I have no positive or negative proof if the recent Encode works with
djgpp. I need more decent environment. iMac with Virtual PC is okay
but it is not exactly mine. I can't use it when my family is awake.
And I found my thinkpad's C: is (Ugh!) Windoze Me so you can't really
run vanilla dos. I want testers on this. Is Laszlo still working on
djgpp? Error logs?
I am exhausted, not just because of 1.50 but also because today happened
to be the day when my government draws the tax from my bank account.
Isn't there something like Open Source Tax Deduction Program ?
Dan the Encode Maintainer
1.50 $Date: 2002/04/19 06:13:02 $
! ! Encode.pm Encode.xs Encode/encoding.h
+ t/fallback.pm
New Fallback API imlemented and documented. See "perldoc Encode"
for details
! lib/Encode/JP/JIS7.pm Encode.pm
+ lib/Encode/PerlIO.pod t/perlio.t
API compliance met. However, it still does not work unless perlio
implements line buffer. See BUGS section in perldoc Encode::PerlIO
As a sensible workaround, perlio_ok() added to Encode.
! encoding.pm
! lib/Encode/Supported.pod
Doc fixes from jhi
Message-Id: <20020418174647(_dot_)J8466(_at_)alpha(_dot_)hut(_dot_)fi>
! CN/CN.pm
Doc fixes from Autrijus
Message-Id: <20020418144131(_dot_)GA10987(_at_)not(_dot_)autrijus(_dot_)org>
! Encode.pm
perlqq mode documented
! t/JP.t
+ t/jisx0201.euc t/jisx0201.ref
! t/jisx0208.euc t/jisx0208.ref
t/JP.t tests more rigorously and with other encodings
t/jisx0201.* added to test JIS7 encodings. jisx0208 is now PURELY
in jis0208 (used to contain jisx0201 part).
! Encode/Makefile_PL.e2x
The resulting Makefile.PL that "enc2xs -M" creates now auto-discovers
enc2xs and encode.h rather than hard-coded. This allows the resulting
module fully CPANizable.
! encoding.pm t/JP.t t/KR.t
PerlIO detection simplified (checks %INC instead of eval{})
! Encode.xs Encode/encode.h
+ Unicode/Makefile.PL Unicode/Unicode.pm Unicode/Unicode.xs
- lib/Encode/Unicode.pm
(en|de)code_xs relocated to where it belongs. Source reindented
to my taste
! bin/enc2xs
Additional (U8 *) cast added as suggested by jhi
Message-Id: <20020417165916(_dot_)A28599(_at_)alpha(_dot_)hut(_dot_)fi>
