[Reposted having sent first copy from generic, unsubscribed address; please
ignore if you've seen the first copy, content here identical.]
Here's an odd one...
When using the "--fetchmailrc -" option to read a config file from stdin,
any stderr output from an explitly-set MDA causes fetchmail to bomb with a
SIGPIPE (and exit with status 2).
Here's a suitably minimal test setup, with only the passwords changed to
protect the guilty:
CONTENTS OF fmrc:
set logfile = "/tmp/fmlog"
set invisible
poll mainline via 0-imap-br1.hpl.hp.com
protocol imap
timeout 150
user sjmz is sjmz
ssl
folder "INBOX.tiny-test"
password NEVER-YOU-MIND-SUNSHINE
# "keep" makes repeated testing cycles easier....
fetchall keep
mda "/tmp/testnoise.sh xyzzy"
expunge 0
CONTENTS OF /tmp/testnoise.sh:
#!/bin/bash
if [[ ${#(_at_)} -gt 0 ]] ; then
echo "What a NOISY little shell script I am." >&2
fi
cat >/dev/null 2>&1
We now issue the command "fetchmail --fetchmailrc fmrc" and see the
following output in /tmp/fmlog:
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: (24406 body octets) What a NOISY little shell script I am.
fetchmail: not flushed
Now, we mutate the commandline to read "fetchmail --fetchmailrc - <fmrc",
and /tmp/fmlog receives the following lines:
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: (24406 body octets) fetchmail: not flushed
fetchmail: SIGPIPE thrown from an MDA or a stream socket error
fetchmail: socket error while fetching from mainline
fetchmail: Query status=2 (SOCKET)
Curious, isn't it? since there's no semantic difference between pointing
--fetchmailrc at a file by name, or giving it that file on stdin with the
--fetchmailrc argument set to -.
If we modify the MDA line of the above fmrc to read
mda "/tmp/testnoise"
so that it no longer produces anything on stderr, the previously
errant command line "fetchmail --fetchmailrc - <fmrc" puts the following
three non-error lines into /tmp/fmlog:
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: (24406 body octets) fetchmail: not flushed
(Note the absence, compared with the first fmlog, of the "What a NOISY
little shell script I am." gubbins.)
It's stderr that matters: if we change the echo command in /tmp/testnoise.sh
to remove the >&2 redirection, and call "fetchmail --fetchmailrc - <fmrc"
again, we see the following four "happy" lines:
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: (24406 body octets) What a NOISY little shell script I am.
fetchmail: not flushed
(Note that the "What a NOISY ..." output appears, but this time it's come
from the MDA's stdout rather than its stderr, so fetchmail croaketh not.)
Platform notes: I see this same behaviour on two separate platforms, both
with quite-new (though not utterly latest-and-greatest: forgive me) versions
of fetchmail. The one on which the above behaviour was discovered is an
HP-UX box running HP-UX 11.11 with fetchmail 6.2.3; for comparison purposes
I also ran a single pair of tests with "--fetchmailrc - <fmrc" versus
"--fetchmailrc fmrc" and the MDA shell-script set to produce Noise on stderr
on a Linux box (RedHat 9, kernel 2.4.20-8, fetchmail 6.2.0). The same
anomolous difference was observed on the RedHat box.
For the terminally curious wondering how I ever fell across this infelicity:
I use "--fetchmailrc -" in order to encrypt my baseline .fetchmailrc, to
keep security-audit people happy that there isn't a permanent copy of the
imap server password, which God help us is also the shell-login password
for that and a couple of other yppasswd-synced boxes. A little shell
script calls openssl with appropriate hard-coded but different password
(yup, this is what Bruce Schneier memorably calls "security theatre" ;-),
and sed magic happens to alter a few critical strings such as the args
to "folder" and "mda" for various related-but-different setups. One of these
is a procmail-testing one, and in one of its variants (which set
VERBOSE=y on the procmail command line) stderr output was being produced,
leading to the SIGPIPE failure. Stripping this back to the small test
case in this message was non-trivial, but hopefully will make finding the
bug (over-eager reuse of file descriptors, maybe?) not too difficult.
For completeness, I'll now append the results of rerunning the two
cases with -v -v added; the non-error "fetchmail -v -v --fetchmail fmrc"
comes first, "fetchmail -v -v --fetchmail - <fmrc" follows. (I've
made one change in this logfile output, substituting the email address
which a spammer had in fact used in the message with an address in
the INVALID.NET domain; as it happens I'm in dispute with that UK-based
spammer and want to continue to be able to assert that the email address
in question has *never* appeared on the Net!)
Hope this all helps to make fetchmail even more robust - it's a damnably
useful tool!
Thanks, Stefek
fetchmail: 6.2.3 querying mainline (protocol IMAP) at Wed May 26 16:26:50 2004:
poll started
fetchmail: mainline key fingerprint:
78:9E:B1:D0:AA:9D:73:6A:FC:5A:7F:4D:A9:7C:11:97
fetchmail: IMAP< * OK Courier-IMAP ready. Copyright 1998-2002 Double Precision,
Inc. See COPYING for distribution information.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4rev1 CHILDREN NAMESPACE
THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT AUTH=PLAIN
fetchmail: IMAP< A0001 OK CAPABILITY completed
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: IMAP> A0002 LOGIN "sjmz" ""
fetchmail: IMAP< A0002 OK LOGIN Ok.
fetchmail: selecting or re-polling folder INBOX.tiny-test
fetchmail: IMAP> A0003 SELECT "INBOX.tiny-test"
fetchmail: IMAP< * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted
\Seen)] Limited
fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * OK [UIDVALIDITY 1051889641] Ok
fetchmail: IMAP< A0003 OK [READ-WRITE] Ok
fetchmail: 1 messages waiting after first poll
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: IMAP> A0004 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 26618)
fetchmail: IMAP< A0004 OK FETCH completed.
fetchmail: IMAP> A0005 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {2212}
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: About to rewrite Return-Path:
<OKPM_1126MO2452_return(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
Rewritten version is Return-Path:
<OKPM_1126MO2452_return(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
fetchmail: About to rewrite From:
"Myoffers"<OKPM_1126MO2452_from(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
Rewritten version is From:
"Myoffers"<OKPM_1126MO2452_from(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
fetchmail: About to rewrite To: EMAIL-ADDRESS-SUPPRESSED(_at_)INVALID(_dot_)NET
Rewritten version is To: EMAIL-ADDRESS-SUPPRESSED(_at_)INVALID(_dot_)NET
fetchmail: About to rewrite Reply-To:
OKPM_1126MO2452_reply(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk
Rewritten version is Reply-To:
OKPM_1126MO2452_reply(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk
fetchmail: about to deliver with: /tmp/testnoise.sh xyzzy
fetchmail: IMAP< )
fetchmail: IMAP< A0005 OK FETCH completed.
fetchmail: IMAP> A0006 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {24406}
fetchmail: (24406 body octets) What a NOISY little shell script I am.
fetchmail: IMAP< )
fetchmail: IMAP< A0006 OK FETCH completed.
fetchmail: not flushed
fetchmail: IMAP> A0007 STORE 1 +FLAGS (\Seen)
fetchmail: IMAP< * 1 FETCH (FLAGS (\Seen))
fetchmail: IMAP< A0007 OK STORE completed.
fetchmail: IMAP> A0008 LOGOUT
fetchmail: IMAP< * BYE Courier-IMAP server shutting down
fetchmail: IMAP< A0008 OK LOGOUT completed
fetchmail: 6.2.3 querying mainline (protocol IMAP) at Wed May 26 16:26:51 2004:
poll completed
fetchmail: not swapping UID lists, no UIDs seen this query
fetchmail: Deleting fetchids file.
fetchmail: normal termination, status 0
fetchmail: Deleting fetchids file.
fetchmail: 6.2.3 querying mainline (protocol IMAP) at Wed May 26 16:26:55 2004:
poll started
fetchmail: mainline key fingerprint:
78:9E:B1:D0:AA:9D:73:6A:FC:5A:7F:4D:A9:7C:11:97
fetchmail: IMAP< * OK Courier-IMAP ready. Copyright 1998-2002 Double Precision,
Inc. See COPYING for distribution information.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4rev1 CHILDREN NAMESPACE
THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT AUTH=PLAIN
fetchmail: IMAP< A0001 OK CAPABILITY completed
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: IMAP> A0002 LOGIN "sjmz" ""
fetchmail: IMAP< A0002 OK LOGIN Ok.
fetchmail: selecting or re-polling folder INBOX.tiny-test
fetchmail: IMAP> A0003 SELECT "INBOX.tiny-test"
fetchmail: IMAP< * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted
\Seen)] Limited
fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * OK [UIDVALIDITY 1051889641] Ok
fetchmail: IMAP< A0003 OK [READ-WRITE] Ok
fetchmail: 1 messages waiting after first poll
fetchmail: 1 message for sjmz at mainline (folder INBOX.tiny-test).
fetchmail: IMAP> A0004 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 26618)
fetchmail: IMAP< A0004 OK FETCH completed.
fetchmail: IMAP> A0005 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {2212}
fetchmail: reading message sjmz(_at_)0-imap-1(_dot_)hpl(_dot_)hp(_dot_)com:1 of
1 (2212 header octets)
fetchmail: About to rewrite Return-Path:
<OKPM_1126MO2452_return(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
Rewritten version is Return-Path:
<OKPM_1126MO2452_return(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
fetchmail: About to rewrite From:
"Myoffers"<OKPM_1126MO2452_from(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
Rewritten version is From:
"Myoffers"<OKPM_1126MO2452_from(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk>
fetchmail: About to rewrite To: EMAIL-ADDRESS-SUPPRESSED(_at_)INVALID(_dot_)NET
Rewritten version is To: EMAIL-ADDRESS-SUPPRESSED(_at_)INVALID(_dot_)NET
fetchmail: About to rewrite Reply-To:
OKPM_1126MO2452_reply(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk
Rewritten version is Reply-To:
OKPM_1126MO2452_reply(_at_)avtg(_dot_)my05(_dot_)co(_dot_)uk
fetchmail: about to deliver with: /tmp/testnoise.sh xyzzy
fetchmail: IMAP< )
fetchmail: IMAP< A0005 OK FETCH completed.
fetchmail: IMAP> A0006 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {24406}
fetchmail: (24406 body octets) fetchmail: IMAP< )
fetchmail: IMAP< A0006 OK FETCH completed.
fetchmail: not flushed
fetchmail: IMAP> A0007 STORE 1 +FLAGS (\Seen)
fetchmail: SIGPIPE thrown from an MDA or a stream socket error
fetchmail: socket error while fetching from mainline
fetchmail: 6.2.3 querying mainline (protocol IMAP) at Wed May 26 16:26:56 2004:
poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: Deleting fetchids file.
fetchmail: normal termination, status 2
fetchmail: Deleting fetchids file.