Re: Procmail looping problem

At 15:21 2003-12-10 -0800, Mike Richards wrote:
> Comments inline.
[snip - lots of snipping, from the dawn of time]

> I'll just clarify a few things that might make this easier. I have a test
> system that is built purely for the purpose of testing this procmailrc
> script.
"test system" means a whole separate computer host, or just a user account, 
or a separate procmailrc setup (such as a sandbox) ?  Since you're 
reluctant to set up a sandbox, I'll assume you probably mean an entirely 
separate computer host.
That's fine - it's certainly trivial to take a backup of an established 
target system and restore it on a spare host, boot it single-user and tweak 
the network settings, then restart it.  That makes for a nice direct 
equivalence for anything you're going to migrate to another system, since 
everything is the same.
> before and after the header re-write.  The first message through does NOT
> contain the header until after formail writes to it.
And, as of yet, nobody on the procmail list has seen anything to 
substantiate this claim.  You're omitting info that you should be LOGGING 
to a logfile, which one would refer to in order to diagnose problems like this.
> Every subsequent "looped" message DOES contain the header but the header 
> is NOT matched by the regexp in the procmailrc to disregard further processing.
And so, if it IS in the message, something is wrong.  Yet, we haven't seen 
the actual headers, or the VERBOSE logfile excerpt, so we all have to take 
you on your word that it really is there, and really is just a single 
_space_ separating it (or that in fact, what you're posting in your emails 
is a space and not intended to be a tab, depending upon what your MUA does 
with such things).
> an example of a fully bloated test procmailrc.
We already saw the procmail recipe you claimed to be using (or at least two 
variants on it).  Still, you haven't provided a fully correlated log 
excerpt or message headers showing the presence of the header in 
question.  I don't keep raising the issue of VERBOSE logging because it's fun.
>    SHELL=/bin/sh
>     LOG="SHELL is $SHELL
>     SENDMAIL is $SENDMAIL"
>     LOGFILE=/var/log/procmaillog
I take it this is an /etc/procmailrc file then?  Joe schmoe user shouldn't 
be writing to /var/log/
>    :0c
>     /var/log/preprocess
FTR, /var/log/ may not be the best place to stuff copies of inbound 
messages, especially when done by a privledged process which is able to 
consume all disk space there if something goes wrong (I dunno, like a mail 
LOOP?).  I realize you're saying this is a test system, but this isn't a 
good practice to get in the habit of for when you set something up on the 
real host - some fragment of braindead code comes along for the ride, and 
your first notification that something is wrong is when you get a call from 
a customer saying their website is down, and the server couldn't log 
anything useful about the problem because the disk was all eaten up.  Write 
your test messages to /tmp/ or better, to a special user directory in 
/home/, which of course should be in a separate filesystem from /, /var, 
and /tmp.
>    :0
>    * ^X-Loop:\ copy
You're relying on a single space.  Please re-read one of my earlier 
messages, where I posted something to the effect of:
        * ^X-Loop:[     ]*unique_tag

The brackets enclose a space and tab.  This allows the script to accomodate 
mail servers which MIGHT mangle the space into something else, or where 
there may be multiple such characters instead of just one.  Mail goes 
through entirely too many crappy mailservers to expect that something won't 
get mangled on the round trip - it's enough to hope that the header will 
still be present...
You're going to kick yourself (if I don't first) if you find that the 
character class fixes your apparent problem, and your problem would have 
been resolved if you'd followed advice instead of skipping around to the 
things you think are more pertinent.
> I am 100% convinced that the regexp is simply failing to match,
And your VERBOSE log sez what?  Surely you're not creating the log and just 
ignoring it?
Certainly, I'd agree, if your VERBOSE log shows that the recipe action 
isn't being run that there's an issue with the regexp matching - say, 
because the data being compared doesn't actually fit the regexp.
> I know that the regexp worked in my sendmail environment but not in my 
> Qmail environment.
A fine indicator that your MTA may be mangling the header.

> I have suspected permissions except that if formail is allowed (and has 
> successfully) written the header information "X-Loop: copy" for me why 
> can't it be matched by procmail?
Toss this in your procmailrc and then CHECK YOUR LOGFILE:

:0c
{
        :0wfh
        | formail -A "X-Loop: unique_tag"

        :0:
        * ^X-Loop:[     ]unique_tag
        tagfound

        :0:
        tagnotfound
}

If that matches, then procmail doesn't have a problem either inserting or 
matching the header. What you have is a header that you're claiming is 
there but really isn't.
If it doesn't match, then, yea, you've got a problem.  I'd start by 
checking the verbose log to see if anything is indicated there, then I'd 
verify that the procmail build is current, and if all else fails, throw 
qmail out the window and return to sendmail, which is the only 100% 
sendmail compatible MTA out there.
> I have looked at the sandbox issue and will get into it if I have to
> although I would prefer not to as I am quite confident of the information
> flowing through procmail based on the carbon copies I make before
Of course there's no need to set up a sandbox and do your own testing if 
you can convince someone else to manually check the mail problem you can't 
sort on your own machine.  Of course, anyone here is going to be doctoring 
up their own test message, and there is only a slight chance that they are 
running qmail, so any tests anyone ELSE does for you are going to be 
limited WRT identifying the problem you're having, and their source 
messages are probably going to match, since they'll be generating them with 
known content.
Manually running the rcfile against a piped-in message containing a known 
header allows you to positively identify that procmail is doing what it is 
supposed to, outside of the influences of your MTA.
> and after processing.  Are you saying that the MTA might be hiding the
> X-Loop header from procmail during that step somehow?
Heck, I don't know - I'm not on your host, I don't use qmail, and when I 
have a problem, I read my VERBOSE logfile and run a recipe in a SANDBOX and 
pipe it suspect messages (as well as manually edited ones which SHOULD 
match the criteria).  It isn't rocket science, and really, when there's a 
ready-made set of scripts sitting on the web to be downloaded, doesn't 
involve a lot of effort or time - the files are even click-and-save, not 
cut-and-paste.
If procmail is writing a file that contains the header, then it was there 
when procmail had the message in memory.  However, you may be visually 
misinterpreting what is in the file - a tab and a space look awfully 
similar under the proper conditions.  A blank line may cause headers to 
prematurely terminate.
> This particular error wasn't new but I have used many different methods of
> doing this but the regexp is the only part that matters and has stayed
> mostly the same.  It is the only part that is consistently failing through
> all methods of trying to get a match.
When you follow up with verbose logs which correlate to the section of the 
rcfile that is suspect, please take a moment and trim the quoted reply text 
to closer context instead of quoting it wholesale.  Thanks.
---
 Sean B. Straw / Professional Software Engineering

 Procmail disclaimer: <http://www.professional.org/procmail/disclaimer.html>
 Please DO NOT carbon me on list replies.  I'll get my copy from the list.


_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail