nmh-workers
[Top] [All Lists]

Re: [Nmh-workers] mhshow/test-charset failures in nmh-1.7 (`Can't convert ?us-ascii to UTF-8')

2017-11-23 04:34:00
Hello Ralph and the entire nmh community,
First thank you very much for the pointers and feedbacks. I was
finally able to find why the test passed and failed, it actually
depends if `libiconv' package (the GNU iconv implementation, NetBSD
has an iconv(3) and iconv(1) also in base)... So, let's try to analyze
that!

For completeness, regarding my `locale' configuration:

 % locale
 LANG="en_US.UTF-8"
 LC_CTYPE="en_US.UTF-8"
 LC_COLLATE="C"
 LC_TIME="en_US.UTF-8"
 LC_NUMERIC="en_US.UTF-8"
 LC_MONETARY="en_US.UTF-8"
 LC_MESSAGES="en_US.UTF-8"
 LC_ALL="en_US.UTF-8"

(But, I think that this shouldn't matter according my code reading
of relevant parts of nmh test suite because the test suite correctly
inject them in the environment.)

Ralph Corderoy writes:
Let us know if you work it out.  That test shouldn't fail regardless of
the local oddness so we'd want to improve, but spotting what's odd
should make diagnosis easier;  that's why I asked.  If you reach a dead
end, tell us that too so one of us can try and work out the problem.

After finding that having the `libiconv' package installed made a
difference I first looked if the several nmh binaries was linked
against the GNU iconv(3) or the NetBSD iconv(3) and in both cases
it's correctly linked to the NetBSD iconv(3).

I have then ktruss(1)-ed the test phases and spotted that iconv(1)
is invoked in test/mhshow/test-charset:

|     23 skip_param_value_checks=0
|     24 if test "$ICONV_ENABLED" -eq 0; then
|     25     text_size=10
|     26 else
|     27     text_size=11
|     28     #### For unknown reasons, the parameter values checks fail on the
|     29     #### FreeBSD10 buildbot.  It doesn't support EBCDIC-US, which is 
used
|     30     #### by the checks, so check for that.  Though that doesn't seem 
to be
|     31     #### the reason.
|     32     printf '\xe4' | iconv -f EBCDIC-US -t UTF-8 >/dev/null 2>&1  ||
|     33         skip_param_value_checks=1
|     34 fi

And, with NetBSD iconv(1) I have:

 % printf '\xe4' | /usr/bin/iconv -f EBCDIC-US -t UTF-8
 U

...while with iconv(1) provided by the `libiconv' package:

 % printf '\xe4' | /usr/pkg/bin/iconv -f EBCDIC-US -t UTF-8
 /usr/pkg/bin/iconv: conversion from EBCDIC-US unsupported
 /usr/pkg/bin/iconv: try '/usr/pkg/bin/iconv -l' to get the list of supported 
encodings
 % echo $?
 1

So, in if GNU iconv(1) is available `$skip_param_value_checks' is
set to 1.

Later, always in test/mhshow/test-charset:

|     85 if test $skip_param_value_checks -eq 1; then
|     86     finish_test
|     87     exit $failed
|     88 fi

This is before 'Encoded parameter value' and 'replacement character
in parameter value' tests, so these last two tests aren't executed,
`$failed' is 0 and hence test-charset PASSes.

I'm now curious if apart FreeBSD and NetBSD with `libiconv' package
installed what happens on other platforms, just checking the exit status
of: 

 $ printf '\xe4' | iconv -f EBCDIC-US -t UTF-8

will be probably enough.

If the exit status is 0 and then, in test-charset context
`$skip_param_value_checks' is 0, what happens if you try (this is
stolen entirely from 'replacement character in parameter value'
test in test-charset):

 $ printf "Subject: invalid parameter value charset\nMIME-Version: 
1.0\nContent-Type: text/plain; charset*=invalid''%%0Dus-ascii\n" | \
 mhshow -file - | cat

Here, I have:

| Subject: invalid parameter value charset
| 
| mhshow: Can't convert ?us-ascii to UTF-8
| mhshow: unable to convert character set from ?us-ascii, continuing...
| [ part  - text/plain -   0B  ]


Thank you for the attention!

-- 
Nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

<Prev in Thread] Current Thread [Next in Thread>