On Wed, Sep 17, 2003 at 5:03:40AM -0400, JoeHill wrote:
What I'm saying is that it seemed my ISP was inappropriately
"tagging" e-mail, ie. adding bits to the header that contained
inappropriate or non-compliant characters, or as you call them,
NULL characters and the like.
I use the term "NULL" with some precision, meaning the char '\0'
hex 0x00, among other things, the terminator of strings in C, etc.
I don't know what characters outside the 7-bit ASCII range between
<space> and '~' are legal in email, but some, like <tab> and <lf>
are, and some like NULL are not. Philip's patch dealt, as I read
it, specifically with NULLs:
+ if ((line[0] == 0) && (linelen > 0))
+ {
...
Until recently, I used an ancient version of fetchmail that failed
badly in the face of NULLs, leaving the following complaint in the log:
'Unable to process From lines (envelopes)'
and refusing to process any further mail until the offending NULLs
were removed. My solution was to call a little script in fetchmail's
preconnect phase that examined the end of fetchmail's log, and if it
found the above error message (from the previous invocation of
fetchmail), it would ssh to my ISP and run a script whose heart
contains:
/usr/ucb/mail -f /var/spool/mail/jimo >/dev/null <<HERE
q
HERE
It turns out that good old Berkeley mail removes NULLs from any
folder you invoke it on, giving you a warning "Ignoring NULLs..."
as it opens up. So this was a cheap, easy hack. Now I'm running
the somewhat-less-old fetchmail 5.9.0, and I haven't logged any
NULL cleanups since its installation. I assumed fetchmail had
been smartened up to cope with this problem. From what I read in
this thread, maybe there's still work to be done.
My ISP runs no Microsoft software at all, so its occasional NULL
problems couldn't be blamed on MS Exchange. I always assumed it
was some sync problem in the network somewhere between the various
mail servers. Whenever I examined NULL-infected mail, the NULLs
were always at the very beginning of the file, at the start of
the leading "From " line, confirming fetchmail's error message above.
Often there'd be 15K NULLs, and sometimes over 100K! Talk about
malformed email... Has anyone seen NULLs elsewhere in an email?
It's curious that I haven't been bothered by a NULL since July,
while some of you folks, with newer versions of fetchmail, are having
trouble. Maybe my ISP coincidentally fixed its network problems.
In any case, as far as patching fetchmail, I'd look for NULLs,
particularly before the start of the header, and eliminate them.
FWIW,
Jim