procmail
[Top] [All Lists]

LINEBUF (RE: More clamav testing (results))

2004-02-14 12:00:19
On Sat, 14 Feb 2004, Gary Funck wrote:

So, neither 'w' nor 'W' are safe in this context. I'd appreciate it if
someone else verified this;

# works fine (subject to LINEBUF limitation)
:0
PIPE=|cat && /bin/false

Note: It seems that both using a pipe and backquotes are subject to the
limitation that the amount of data being copied into the variable must
be less than LINEBUF in length.

This should work.  This should also NOT be subject to LINEBUF -- LINEBUF
applies only to lines *read from the rcfiles*, which includes all forms of
*expansions* but not *assignments*.

That is, PIPE above will only be truncated at the time you refer to it as
$PIPE.  So this should be OK:

        :0
        ASSIGN=|zsh -fc 'repeat 100000 print 1 megabyte'

And in fact it is ("Score:" below is from "* 1^1 ASSIGN ?? ."):

----------
procmail: Assigning "ASSIGN="
procmail: [15949] Sat Feb 14 10:04:03 2004
procmail: Executing "zsh,-fc,repeat 100000 print 1 megabyte"
procmail: [15949] Sat Feb 14 10:04:06 2004
procmail: Assigning "LOG=PROCMAIL_OVERFLOW is 
"
PROCMAIL_OVERFLOW is 
procmail: [15949] Sat Feb 14 10:04:07 2004
procmail: Score: 1000000 1000000 "."
procmail: Assigning "LOG=ASSIGN is 1000000 bytes
"
ASSIGN is 1000000 bytes
----------

But this will get truncated and set PROCMAIL_OVERFLOW:

        EXPAND="$ASSIGN"

And so will this (backticks are expansion):

        ASSIGN=`zsh -fc 'repeat 100000 print 1 megabyte'`

----------
procmail: [15945] Sat Feb 14 10:03:07 2004
procmail: Executing "zsh,-fc,repeat 100000 print 1 megabyte"
procmail: [15945] Sat Feb 14 10:03:07 2004
procmail: Excessive output quenched from "zsh"
procmail: Assigning "PROCMAIL_OVERFLOW=yes"
----------

This presents a problem, because as Dallman mentioned the pipe-capture
assignments are unsafe on some platforms/compilations.  You can get an
unofficial version of procmail with a fix for that specific bug from
<http://www.well.com/user/barts/email/procmail-3.23pre.tar.gz>.  (That
source unfortunately does not solve the mysterious dropped-"F" bug.)

The workaround for that, assuming that what you want to capture in a
variable is the message source, is:

        :0
        * 2^1 HB ?? .
        { LINEBUF = $= }

        LOG="LINEBUF set to $LINEBUF$NL"

        # LINEBUF is now set to twice the length of the message, so:
        ASSIGN="`cat`"

        :0
        * 1^1 ASSIGN ?? .
        { LOG="ASSIGN is $= bytes$NL" }

----------
LINEBUF set to 2000000
ASSIGN is 1000000 bytes
----------

(I didn't set VERBOSE there because the output of `cat` is expanded before
being logged (unlike the pipe-capture case) and that, obviously, produces
a megabyte of log output I didn't need.)

I'm uncertain as to whether the actual restriction is (LINEBUF-1)
characters (to allow for an internal null character, perhaps, and so
on).

It may be as much as LINEBUF-1 bytes, but it's not guaranteed.  For 
example, if I change 2^1 to 1^1 in the recipes above:

LINEBUF set to 1000000
procmail: Excessive output quenched from "cat"
ASSIGN is 909083 bytes

I'm also uncertain how procmail behaves if the data being read contains
control characters like a null byte (\000), and/or control-z on Windows
platforms.

Control-Z on windows wouldn't be an issue, but I'm not sure about null
either.  I *believe* procmail would capture the entire output, but I
don't know wheter it could output it correctly e.g. in a LOG=.  Procmail
does handle binary *message content* correctly.

The LINEBUF restriction is serious, and surprising, and might be the
cause of problems that we've had with the virus scanning example.

In the mythical procmail of the future, I think that all fixed LINEBUF
restrictions should be removed.

I don't think having a limitation on the size of lines expanded from the 
rcfile is all that serious a restriction.  And it is all documented, so
it shouldn't have been especially surprising.

Left capturing is really needed. I'd settle for delimited capturing:

* VIRUS ?? ' : \/\<.*\>() FOUND$

where () is a null pattern match that serves to delimit the match
beginning at \/.

That won't be satisfactory because it makes it unpredictable what occurs
when you construct a composite pattern from several variable expansions,
each of which might have \/ and/or () [or any other chosen delimiter].

There's no way to introduce a new set of delimiters into procmail regex
without breaking somebody's existing usage.  Left-capture is about the
only possible approach that wouldn't break something, and even that would
have to be marked with some new syntax outside the regex, probably a new
recipe flag, along the lines of the "D" flag for case-sensitivity.


_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail