Bennett Todd <bet(_at_)newritz(_dot_)mordor(_dot_)net> writes:
procmail by design pulls the entire message into virtual memory, which means
it can wreck the machine if you try and use it to pass gigantic messages.
Solutions divide into three categories that I can think of:
1) Ameliorate the problem --- buy loads of RAM, allocate huge tracts of VM
(i.e. swap), then raise the per-process memory allocation limits. Also go
for later versions of procmail, one thing they've tried to work on is a
category of bad interaction with certain malloc implementations that could
double memory requirements. None of these will solve your problem, they'll
just make it harder to hit. Some clever bastard decides to email a zip of
their entire hard drive as an attachment (backups are easy!) and you're
still gonna be hurting.
Heh, like the student who email an entire CD to himself? While a
mailserver should have plenty of memory just so that it can handle a
good workload, this will not, as you point out, 'solve' the problem.
2) Fix the problem _Completely_ --- configure your MTA (sendmail, qmail,
Postfix, whatever) to discard messages greater than some reasonable size. I
recommend 64KB.
64KB seems way low. I recommend at least 10MB, if not more. You
better have at least 64MB on your machine though.
3) Ameliorate the problem a lot better than the choices in (1) --- if you were
to switch to Maildrop[1], then your message size limit would be determined
by available tmp file space, rather than by available VM, and with careful
filesystem layout this can be made equivalent to available local delivery
space, which means the problem goes away altogether. Maildrop is not
Assuming you mean to use the same filesystem for temp space and local
delivery, that would limit you to half the space remaining, so it's
still possible to receive a 'killer message', one that you can feed to
the delivery agent but not deliver. (Putting temp space on a separate
filesystem doesn't stop killer messages unless you somehow mirror free
space between them.)
I actually have code which makes procmail read messages into a temp
file which is then mmap()ed, however there's a real hornet's nest of
catches which such a plan, the biggest problem of which is "where do
you put the temp file?" Remember: it should be larger than the largest
legal message (from the limit set by (2) above), world writable
(procmail is usually not running as root), and not NFS mounted. I used
/var/tmp when I was playing the patch, but that's not a good choice on
many systems. If I ever integrate this code into the main line of
procmail, it'll not be enabled by default because there's no way for
procmail's autoconf script to figure out what that directory should be
-- it might not even exist on the system where procmail is being
compiled.
Now, answer me this: if I have an X MB file system to use as temp
space, why shouldn't I just tell the OS to swap to that area and keep
everything in memory? How is the temp file solution going to be any
more efficient? (If your answer is that the temp file solution allows
the space to be used for other program's temp files as well, then I'll
tell you to mount a ramfs/swapfs on that directory and try again.)
Using a temp file is also bleeding inefficient for small messages. To
paraphrase Rob Pike, "complex algorithims are efficient only when N is
large; N is usually small." I hope maildrop is smart enough to only
dump to a temp file when the message goes over some limit.
Replies to procmail-dev, please.
Philip Guenther