procmail
[Top] [All Lists]

Re: holy benchmarks, Batman

2003-03-31 10:47:16
On 30 Mar, Jeff Orrok wrote:
| I have a fairly involved suite of procmailrc's for a client that processes
| all of the email leads that he receives from visitors to various websites.
| Everything seems to be working ok, but today when I was testing in preparation
| for upgrading, I was horrified to discover that it was taking almost 8 minutes
| to process 8 messages in my test file.  I'm not sure when it started doing
| this, since I proceeded to regress my changes, and even pulled his current 
copy
| over for comparison, and every version seems to be doing the same thing.
| 
| I can't believe that anything I'm doing would take up that much time, but I
| guess I've got to face the music.  So can anyone point me in the direction of
| some performance analysis utilities that I can use to track down my problem?
| 

My first thought was a locking problem, but your description doesn't
correspond to default LOCKTIMEOUT.  If you use any regional lockfiles,
and especially if you modify LOCKTIMEOUT, that would be one place to
look.  If procmail has to force removal of a stale lockfile, that should
show in the log.

If that isn't it, I'd look first at any external programs that are
called.  Does this involve database updates?  Could there be a locking
problem with a database?  Are you calling any programs or using any
code which you haven't written?  If you don't have intimate
understanding of what they're doing, that would be a good place to
start. Before and after any suspect recipes, log an identifier and the
time to try and isolate the offender(s).  Since it's taking a
ridiculous amount of time anyway, a few additional forks to run date
won't hurt.

FWIW, I've been playing around with timing message delivery for some
time now. One of these days I'll make it available but, right now, it's
just too ugly.  For sure, this would not be intended for a system where
resource use is an issue, but the systems are my own so I've gone a
little overboard for curiosity's sake.  Your message contained this
header (inserted here):

X-RCstops: 1.05 53 (0.6973) / 31 (0.6874) last: localrc [11010]

That's the machine load when /etc/procmailrc starts, the count and
elapsed time of "system" rcfiles processed, the count and time of
personal rcfiles, the last rcfile and the pid.  The rcfile counts are
skewed higher because of some goofy stack variables I keep for each
rcfile, but it's illustrative. I haven't compiled any of the statistics
I've captured, but seat of the pants analysis says this message is
representative to a little high.  I wouldn't say I'm profligate
regarding resource use. I optimize for fun, but also do what I want
(sometimes for fun), and am not adverse to wielding the perl sledge
hammer. ;-)

The rcfiles include virus checking, rigorous routines to identify local
and list mail, checking whitelists, removal of moron disclaimers and
ridiculous signatures (especially ascii art) and list footers, keeping
stacks of VERBOSE and LOGFILE values to allow changing those values per
rcfile/recipe and having them automagically revert to their incoming
values through multiple levels of INCLUDERC's, and more. The point is,
most messages go through a lot of stuff and probably average somewhere
between .9 to 1.4 seconds (under normal load - dual PII 450 w/ 512M ram,
running Linux).  This includes the overhead of all this crap and
insertion of headers to document it all.  I'd say you do have something
running amok. ;-)


-- 
Email address in From: header is valid  * but only for a couple of days *
This is my reluctant response to spammers' unrelenting address harvesting



_______________________________________________
procmail mailing list
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail

<Prev in Thread] Current Thread [Next in Thread>