procmail
[Top] [All Lists]

Re: Backup in mbox format?

2000-01-16 17:05:09
On Jan 16,  2:30pm, era eriksson wrote:

The problem is that it is rather expensive to remove old messages from
the beginning of an mbox file, while deleting the oldest files in a
directory is not much work at all.

If you don't expect the backup archive to grow very large, or don't
expect to use it very often, or if you can live with a solution which
is cheap in more ways than one :-) I would recommend to simply append
to an mbox file and rename it from cron at midnight. You should find
this explained in more detail using the search terms above.

I had a similar query recently, here's the thread:
http://www.xray.mpe.mpg.de/mailing-lists/procmail/1999-12/msg00130.html

I've gone with the strategy outlined in that msg., running a
script in a cronjob each day that looks at each of the various
mbox files where I save either mailing list messages or messages
that relate to particular topics, along with my inbox, and
outgoing saved messages.   Although not terribly efficient,
since the mboxes being moved are by definition not very large
(due to the pruning process), the whole procedure finishes
within a few minutes.

The following script is admittedly a little bit of overkill,
but does the job.  It assumes that mail is being saved in
the directory ~/mail and the mboxes are named *-mail.  When
an mbox has more that $maxmsgs it is trimmed back to $minmsgs
(200 and 100 respectively).  The excess messages are appended
to an mbox of similar name, in a directory named ~/mail/archive.

The outbox (~/.sent) and inbox ($MAIL) files are archived
differently.  Excess messages are first saved to files called 
~/old_mail/sent.save and ~/old_mail/inbox.save.  When
these files get to be too big (2000 messages), they are
moved to a file called ~/old_mail/sent.YYYYMMDD and 
~/old_mail/inbox.YYYYMMDD, which are date stamped by
the current date.

Some special handling is required to make sure that lockfiles
are used to keep the process from colliding with incoming
procmail filters.  The filters obviously neeed to use lcokfiles.
For example, the procmail list filter looks like this
(the :0: tells procmail to use a lockfile named
procmail-mail.lock below):

    #
    # Procmail mailing list
    #
    :0:
    * ^X-Mailing-List:.*procmail
    procmail-mail

Other things to look out for are to properly set the permissions
and group for the inbox and other files, and to set the
last mod. date on the updated mail file to agree with the old one.

The rotate_mail csh script:

#!/bin/csh -f
set dt = `date +'%Y%m%d'`
set path = (. /usr/local/bin $path )
if (! $?MAIL) setenv MAIL /usr/mail/$USER
umask 77
onintr cleanup
set maxmsgs = 200
set minmsgs = 100
# rotate mail folders when they get too big
cd ~/mail
#
#  Archive all mail files not modified in past 28 days
#
set flist = (`find *-mail -mtime +28 -print`)
foreach f ($flist)
    set lockfile = ${f}.lock
    lockfile -s 10 -r 6 $lockfile
    if ($status != 0) then
        continue
    endif
    set arc = archive/${f}
    if (-e $arc) then
       echo "" >> $arc
    endif
    cat $f >> $arc
    touch -r $f $arc
    chmod 600 $arc
    rm -f $f
    rm -f $lockfile
end
#
#  Archive all mail with more than $maxmsgs, and trim back to $minmsgs
#
foreach f (*-mail)
    set lockfile = ${f}.lock
    lockfile -s 10 -r 6 $lockfile
    if ($status != 0) continue
    set nmsgs = `formail -X 'From ' -s < $f | wc -l`
    if ($nmsgs > $maxmsgs) then
        set arc = archive/${f}
        if (-e $arc) echo "" >> $arc
        @ cnt = $nmsgs - $minmsgs
        formail -$cnt -s < $f >> $arc
        formail +$cnt -s < $f > ${f}.tmp
        touch -r ${f} ${f}.tmp
        mv -f ${f}.tmp ${f}
        chmod 600 ${f} $arc
    endif
    rm -f $lockfile
end
#
#  move old messages in .sent file to ~/old_mail/sent.save
#  and when that file exceeds 2000 messages move it to ~/old_mail/sent.yyyymmdd
#
set f = ~/.sent
set arc = ~/old_mail/sent.save
set old = ~/old_mail/sent.${dt}
if (-e $f) then
    set lockfile = ${f}.lock
    lockfile -s 10 -r 6 $lockfile
    if ($status == 0) then
        set nmsgs = `formail -X 'From ' -s < $f | wc -l`
        if ($nmsgs > $maxmsgs) then
            if (-e $arc) echo "" >> $arc
            @ cnt = $nmsgs - $minmsgs
            formail -$cnt -s < $f >> $arc
            formail +$cnt -s < $f > ${f}.tmp
            touch -r ${f} ${f}.tmp
            mv -f ${f}.tmp ${f}
            chmod 600 ${f} $arc
            rm -f $lockfile
            set nmsgs = `formail -X 'From ' -s < $arc | wc -l`
            if ($nmsgs > 2000) mv $arc $old
        endif
    endif
    if (-e $lockfile) rm -f $lockfile
endif
#
#  move old messages in /usr/mail/user file to ~/old_mail/inbox.save
#  and when that file exceeds 2000 messages move it to ~/old_mail/inbox.yyyymmdd
# 
lockfile -ml
if ($status != 0) exit 2
set f = "$MAIL"
set arc = ~/old_mail/inbox.save
set old = ~/old_mail/inbox.${dt}
if (-e $f) then
    set nmsgs = `formail -X 'From ' -s < $f | wc -l`
    if ($nmsgs > $maxmsgs) then
        if (-e $arc) echo "" >> $arc
        @ cnt = $nmsgs - $minmsgs
        formail -$cnt -s < $f >> $arc
        formail +$cnt -s < $f > ${f}.tmp
        touch -r ${f} ${f}.tmp
        mv -f ${f}.tmp ${f}
        chmod 600 $arc
        chgrp mail ${f}
        chmod 660 ${f}
        lockfile -mu
        set nmsgs = `formail -X 'From ' -s < $arc | wc -l`
        if ($nmsgs > 2000) mv $arc $old
    endif
endif
if (-e ${MAIL}.lock) lockfile -mu
#
# move mail-log to the archive directory, when it gets too big.
#
if (-e mail-log) then
    lockfile -ml
    if ($status == 0) then
        if (`wc -l < mail-log` > 1500) mv -f mail-log archive
    endif
    if (-e ${MAIL}.lock) lockfile -mu
endif
#
#  Send a message to make sure mail box isn't empty
#
if (-z "$MAIL") then
    date | Mail -s "rotate_mail completed" $USER
endif
#
# Unlock the mail spool file
#
cleanup:
if ($?lockfile && -e $lockfile) rm -f $lockfile
if (-e ${MAIL}.lock) lockfile -mu

<Prev in Thread] Current Thread [Next in Thread>