There is currently a general problem with both procmail and SmartList:
there is no regression testing (other than us users).
I've been holding off on migrating from 3.10 to 3.11 for both procmail
and SmartList because of this lack. Although I trust Stephen to have
done a thorough, rigorous testing of his software, there is no way he
can test in our specific, mailer environment. Thus, we need to run some
tests to assure ourselves that the new version at least doesn't break
existing configurations.
It would be good if procmail and SmartList had some regression tests
with which we could compare the new version with the old version.
So, I've just written a shell script, runtests, with which we can begin
constructing a set of regression tests. There are a couple of remedial
regression tests already, but nothing significant enough with which we
can say: "yup! the new procmail/SmartList works just fine, and doesn't
break anything!".
Please review and make comments, but, most importantly, contribute some
tests and improve the regression testing.
The script, runtests, is currently designed only as a procmail
regression tester. I'm certain that other feature will be needed,
perhaps another script to enable SmartList regression testing. In
particular, what features would be necessary in order to run "flist" in
a "sandbox" environment, where both the input and output can be verified
without having real mail get generated in the process?
When writing tests, be sure to test for both failure and success. The
actual output of any particular test doesn't matter, as long as it is
consistent and doesn't vary over time.
The script "runtests" is self-documenting (runtests -h), and tries to
avoid comparing test output which varies over time (such as the logfile
output with the procmail process id and ctime string).
Shar archive follows.
BTW, Happy New Year!
_______________________________________
Alan K. Stebbens <aks(_at_)hub(_dot_)ucsb(_dot_)edu>
College of Engineering
University of California, Santa Barbara
============================= cut here ===================================
#!/bin/sh
# This is a shell archive (produced by shar 3.49)
# To extract the files from this archive, save it to a file, remove
# everything above the "!/bin/sh" line above, and type "sh file_name".
#
# made 12/30/1995 04:34 UTC by aks(_at_)dokoka(_dot_)ucsb(_dot_)edu
# Source directory /fs/eci1c/home/staff/aks/src/mail/procmail/procmail-lib
#
# existing files will NOT be overwritten unless -c is specified
#
# This shar contains:
# length mode name
# ------ ---------- ------------------------------------------
# 10259 -r-xr-x--x runtests
# 90 -rw-r--r-- t/rc/rc.1-1
# 949 -rw-r--r-- t/rc/rc.1-2
# 151 -rw-r--r-- t/in/in.1
# 801 -rw------- t/in/in.2
#
if touch 2>&1 | fgrep 'amc' > /dev/null
then TOUCH=touch
else TOUCH=true
fi
# ============= runtests ==============
if test -f 'runtests' -a X"$1" != X"-c"; then
echo 'x - skipping runtests (File already exists)'
else
echo 'x - extracting runtests (Text)'
sed 's/^X//' << 'SHAR_EOF' > 'runtests' &&
#!/bin/sh
# runtests -- procmail regression tester
#
# Author: Alan K. Stebbens
# $Id: runtests,v 1.1 1995/12/30 04:34:41 aks Exp $
#
# runtests [options] [tests... ]
#
# If neither -O or -N are given, assumes that a regression test against
# the default procmail path is desired.
#
# Assumes: there is a directory named "t", in which the following
# subdirectories exist:
#
# in - contains STDIN files, numbered serially: in.1, in.2, ...
#
# rc - contains the recipe test files, numbered by
# serial number for corresponding input file, and by
# test number: rc.1-1, rc.1-2, rc.1-3, rc.2-1, rc.2-2, rc.2-3, ...
#
# out - default directory for output files: named with "out.", "log.",
# and "mail." as prefixes, and the test numbers as suffixes. Eg:
# for the rcfile: rc.3-4, the following output files may appear
# in the "out" directory: out.3-4, log.3-4, mail.3-4
#
# old - contains the output files produced by the "old" version of
# procmail, when given "-O".
#
# new - contains the output files produced by the "new" version of
# procmail, when given "-N".
#
# ref - contains the "reference" subdirectories: "out", "log",
# and "mail", produced when given the "-c" option.
#
# diffs - contains the differences files of all the output
# subdirectories.
X
usage () {
X if [ "$1" != "" ]; then
X echo "$*" 1>&2
X fi
X cat <<EOF 1>&2
usage: $0 [options] [ tests ... ]
Run procmail on one or more regression tests, with corresponding input
files, and comparing the resulting output files. A new version of
procmail may be validated using this script and a set of regression
tests.
X
Options:
X -D directory
X set the regression test directory path [t]
X -O oldprocmail
X set path to old procmail
X -N newprocmail
X set path to new procmail
X -C create reference output using new or default procmail
X -l list the possible tests
X -n do not remove log diffs files, even when no changes
X -d debug
X -v Perform procmail tests with VERBOSE=yes
X
The regression test directory, "t" by default, must contain, at the very
least, the directories "rc" and "in". These two directories contain
recipe test files and input files, respectively. The recipe files are
named in the pattern "rc.NN-MM", where NN must select a correspondingly
named input file in "in/in.NN", and "MM" is a serial number. The output
files are named "out.NN-MM", "log.NN-MM", and "mail.NN-MM",
corresponding to the recipe file name.
X
When given an old or new path to procmail, the directories "old" and
"new" are used to contain the corresponding output files; otherwise,
"out" and "ref" are used. The output files in "ref" are created only
when given the "-C" option.
X
The directory "diffs" contains any differences discovered between the
output files in the out/ref or old/new directories. There are some
insignificant differences in the "log.NN-MM" files, where procmail
prints its process id and ctime string, which are ignored when
determining whether or not real changes have occured. The diffs files
are removed when empty. The "log.NN-MM" files are removed even when
non-empty if they contain nothing but the insignificant changes, unless
the "-n" option is given.
X
EOF
X
X exit 1
}
X
# Small subroutine to run a test with the output into a particular
# directory
#
# runtest TESTNUM TESTDIR [procmailpath] [path]
#
X
runtest () {
X if [ $debug -gt 0 ]; then
X set -x
X fi
X testnum=${1:?'Missing test number'}
X outdir=${2:?'Missing test directory'}
X procmailpath=${3:-$procmail}
X addpath=${4:-''}
X # Each test must have a recipe file -- otherwise, what's the purpose?
X rcfile=rc.$testnum
X if [ ! -f rc/$rcfile ]; then
X echo "Test $testnum has no test recipe file! Skipped."
X return 1
X fi
X
X # Make a link to the rcfile so procmail can read it, after it changes
X # to $MAILDIR
X if [ ! -f $outdir/$rcfile ]; then
X ln rc/$rcfile $outdir/.
X fi
X
X # And, each test must have an input file -- otherwise, nothing to do!
X in=in/in.`echo $testnum | sed -e 's/-.*$//'`
X if [ ! -f $in ]; then
X echo "Test $testnum has no input file! Skipped."
X return 1
X fi
X # Now create the output file names
X out=$outdir/out.$testnum ; cat /dev/null >$out
X log=$outdir/log.$testnum ; cat /dev/null >$log
X mail=$outdir/mail.$testnum ; cat /dev/null >$mail
X echo "Running test $testnum, with output files in $outdir ..."
X if [ $debug -gt 0 ]; then
X echo "\
X $procmailpath -m \
X TESTNUM=$testnum \
X MAILDIR=$outdir \
X DEFAULT=mail.$testnum \
X LOGFILE=log.$testnum \
X LOGABSTRACT=all \
X SENDMAIL=sendmail.sh \
X VERBOSE=$verbose \
X $rcfile <$in >$out 2>$log" | tr -cs "[ -~]" "[ *]" ; echo ""
X fi
X if [ $debug -ne 1 ]; then
X fixpath=$PATH
X if [ -n "$addpath" ]; then
X fixpath=$addpath:$PATH
X fi
X ( PATH=$fixpath ; export PATH ;
X $procmailpath -m \
X TESTNUM=$testnum \
X MAILDIR=$outdir \
X DEFAULT=mail.$testnum \
X LOGFILE=log.$testnum \
X LOGABSTRACT=all \
X SENDMAIL=sendmail.sh \
X VERBOSE=$verbose \
X $rcfile <$in >$out 2>$log )
X # There is some asynchronicity going on .. procmail output doesn't get
X # flushed all the time
X sleep 1
X status=$?
X if [ $status != 0 ]; then
X echo "Test $testnum: $procmailpath returned status $status"
X fi
X else
X return 1
X fi
X return 0
}
X
# compare TEST OLDDIR NEWDIR
#
# Compare the results of TEST using OLDDIR versus NEWDIR
X
compare () {
X if [ $debug -gt 0 ]; then
X set -x
X fi
X testnum=${1:?'Missing test number'}
X olddir=${2:?'Missing old directory'}
X newdir=${3:?'Missing new directory'}
X for dir in $olddir $newdir; do
X if [ ! -d $dir ]; then
X echo "Directory $dir non-existant; cannot compare."
X return 1
X fi
X done
X # Check all the output files
X firstmsg="Comparing test $testnum output: "
X bad=''
X for kind in out log mail; do
X out=$kind.$testnum
X old=$olddir/$out
X new=$newdir/$out
X if [ -f $old -a -f $new ]; then
X if [ -n "$firstmsg" ]; then
X echo -n "$firstmsg"
X firstmsg=
X fi
X echo -n $kind
X dif=diffs/$out ; cat /dev/null >$dif
X diff $old $new >$dif 2>&1
X # If the file is a logfile, don't match on the procmail process
X # id's, since they change
X if [ $kind = log ] && \
X egrep -s -v '^((< |> )procmail: \[[0-9]+\]|---|[0-9,]+[adc][0-9,]+)'
$dif ||
X [ $kind != log -a -s $dif ]; then
X echo -n ":nope"
X bad="$bad $dif"
X else
X if [ -z "$leavediffs" ]; then
X rm -f $dif
X fi
X echo -n ":okay"
X fi
X echo -n ", "
X elif [ -f $old ]; then
X echo "$new doesn't exist"
X elif [ -f $new ]; then
X echo "$old doesn't exist"
X fi
X done
X if [ -n "$bad" ]; then
X echo "; diffs in file(s) $bad"
X else
X echo "; no diffs."
X fi
}
X
#######################
# Start processing here
X
debug=0
verbose=no
X
# Check for options
while test $# -gt 0 ; do
X case "$1" in
X -D*) testdir=`echo $1|sed -e 's/^-D//'`
X if [ -z "$testdir" ]; then
X if [ $# -gt 1 ]; then
X testdir=$2
X shift
X else
X usage "Missing directory after -D"
X fi
X fi
X if [ ! -d $testdir ]; then
X echo "Directory $testdir doesn't exist!"
X exit 1
X fi
X if [ ! -d $testdir/rc -o ! -d $testdir/in ]; then
X echo "Directory $testdir does not have an \"rc\" or \"in\"
subdirectory!"
X exit 1
X fi
X ;;
X -O*) oldprocmail=`echo $1|sed -e 's/^-O//'`
X if [ -z "$oldprocmail" ]; then
X if [ $# -gt 1 ]; then
X oldprocmail=$2
X shift
X else
X usage "Missing procmail path after -O"
X fi
X fi
X if [ ! -f $oldprocmail ]; then
X echo "The path $oldprocmail does not exist!"
X exit 1
X fi
X if [ ! -x $oldprocmail ]; then
X echo "The path $oldprocmail is not executable!"
X exit 1
X fi
X ;;
X -P*) newprocmail=`echo $1|sed -e 's/^-P//'`
X if [ -z "$newprocmail" ]; then
X if [ $# -gt 1 ]; then
X newprocmail=$2
X shift
X else
X usage "Missing procmail path after -P"
X fi
X fi
X if [ ! -f $newprocmail ]; then
X echo "The path $newprocmail does not exist!"
X exit 1
X fi
X if [ ! -x $newprocmail ]; then
X echo "The path $newprocmail is not executable!"
X exit 1
X fi
X ;;
X -C) createref=1 ;;
X -dd) debug=2;;
X -d) debug=1;;
X -v) verbose=yes;;
X -n) leavediffs=1;;
X -l) listtests=1;;
X -help|-hel|-he|-h) usage ;;
X -*) usage "Unknown option: $1" ;;
X *) break ;;
X esac
X shift
done
X
if [ -z "$testdir" ]; then
X for dir in t ../t none ; do
X if [ -d $dir/in -a -d $dir/rc ]; then
X break
X fi
X done
X if [ $dir = none ]; then
X echo "Can't find the regression test directory!"
X exit 1
X fi
X testdir=$dir
fi
X
# Set the working directory to the regression test directory
cd $testdir
echo "Running tests from directory $testdir..."
maildir=.
X
if [ -n "$listtests" ]; then
X echo "The following tests are available: "
X ls -C rc | sed -e 's/rc\./ /g'
X exit 1
fi
X
X
if [ -n "$oldprocmail" ]; then
X oldpath=`dirname $oldprocmail`
else
X oldprocmail=procmail
fi
if [ -n "$newprocmail" ]; then
X newpath=`dirname $newprocmail`
else
X newprocmail=procmail
fi
if [ "$oldprocmail" != "$newprocmail" ]; then
X newversion=1 dirs='new old'
else
X dirs='out ref'
fi
X
# Scan test arguments
if [ $# -eq 0 ]; then
X tests=`(cd rc; echo * | sed -e 's/rc\.//g')`
X case "$tests" in
X '*') echo "No test recipe files!" ; exit 1;;
X esac
else
X tests="$@"
fi
X
# Housekeeping -- make sure the output directories are setup
for kind in $dirs diffs ; do
X if [ ! -d $kind ]; then
X echo "Creating $kind subdirectory.."
X mkdir $kind
X fi
done
X
# Now run the tests
for test in $tests ; do
X # Allow for NN or NN-MM as test arguments
X case $test in
X *-*) testgroup=$test ;;
X *) testgroup=`(cd rc; echo rc.$test-* | sed 's/rc\.//g')`
X case "$testgroup" in
X "$test-*")
X if [ -f rc/rc.$test ]; then
X testgroup=$test
X else
X testgroup=
X fi ;;
X esac ;;
X esac
X if [ -z "$testgroup" ]; then
X echo "No such test as $test"
X continue
X fi
X for tnum in $testgroup ; do
X if [ -n "$newversion" ]; then
X runtest $tnum old $oldprocmail $oldpath || continue
X runtest $tnum new $newprocmail $newpath || continue
X compare $tnum old new
X elif [ -n "$createref" ]; then
X runtest $tnum ref $newprocmail $newpath
X else
X runtest $tnum out procmail $newpath || continue
X compare $tnum ref out
X fi
X done
done
exit
SHAR_EOF
$TOUCH -am 1229203495 'runtests' &&
chmod 0551 runtests ||
echo 'restore of runtests failed'
Wc_c="`wc -c < 'runtests'`"
test 10259 -eq "$Wc_c" ||
echo 'runtests: original size 10259, current size' "$Wc_c"
fi
# ============= t/rc/rc.1-1 ==============
if test ! -d 't'; then
echo 'x - creating directory t'
mkdir 't'
fi
if test ! -d 't/rc'; then
echo 'x - creating directory t/rc'
mkdir 't/rc'
fi
if test -f 't/rc/rc.1-1' -a X"$1" != X"-c"; then
echo 'x - skipping t/rc/rc.1-1 (File already exists)'
else
echo 'x - extracting t/rc/rc.1-1 (Text)'
sed 's/^X//' << 'SHAR_EOF' > 't/rc/rc.1-1' &&
# This is the simplest regression test -- do nothing
# except drop the mail into $DEFAULT
SHAR_EOF
$TOUCH -am 1229194795 't/rc/rc.1-1' &&
chmod 0644 t/rc/rc.1-1 ||
echo 'restore of t/rc/rc.1-1 failed'
Wc_c="`wc -c < 't/rc/rc.1-1'`"
test 90 -eq "$Wc_c" ||
echo 't/rc/rc.1-1: original size 90, current size' "$Wc_c"
fi
# ============= t/rc/rc.1-2 ==============
if test -f 't/rc/rc.1-2' -a X"$1" != X"-c"; then
echo 'x - skipping t/rc/rc.1-2 (File already exists)'
else
echo 'x - extracting t/rc/rc.1-2 (Text)'
sed 's/^X//' << 'SHAR_EOF' > 't/rc/rc.1-2' &&
# Test some basic features of procmail
#
DELIVERED=yes # cause the mail to succeed
X
# Here's the input file
# From aks Fri 22 May 92
# To: aks(_at_)hub
# Subject: test:aa
# Date: Fri, 22 May 92 16:11:08 +22318216
# From: Joe bloe <root(_at_)anywhere>
#
# This is a test- - ignore.
X
:0
* ^Foobar: noway
{ LOG="BAD: Foobar matched
" }
:0E
{ LOG="Good: else condition works!
" }
X
:0
* ^To:
{ LOG="Good: ^To: matched
" }
:0E
{ LOG="BAD: ^To: failed to match
" }
:0A
{ LOG="Good: and condition clause okay
" }
:0E
{ LOG="BAD: and condition clause failed
" }
X
:0
* ^TOaks
{ LOG="Good
" }
X
:0E
{ LOG="BAD: ^TOaks failed
" }
X
:0
* ^Subject: test
{ LOG="Good
" }
X
:0E
{ LOG="BAD: ^Subject: test failed
" }
X
:0
* ^subject: test
{ LOG="Good
" }
:0E
{ LOG="BAD: ^subject: test failed
" }
X
:0D
* ^subject: test
{ LOG="BAD: ^subject matched but shouldn't
" }
:0E
{ LOG="Good
" }
X
:0
* ^From:.*root(@|[, ]|$)
{ LOG="Good
" }
:0E
{ LOG="BAD: ^From:.*root match failed
" }
SHAR_EOF
$TOUCH -am 1229203395 't/rc/rc.1-2' &&
chmod 0644 t/rc/rc.1-2 ||
echo 'restore of t/rc/rc.1-2 failed'
Wc_c="`wc -c < 't/rc/rc.1-2'`"
test 949 -eq "$Wc_c" ||
echo 't/rc/rc.1-2: original size 949, current size' "$Wc_c"
fi
# ============= t/in/in.1 ==============
if test ! -d 't/in'; then
echo 'x - creating directory t/in'
mkdir 't/in'
fi
if test -f 't/in/in.1' -a X"$1" != X"-c"; then
echo 'x - skipping t/in/in.1 (File already exists)'
else
echo 'x - extracting t/in/in.1 (Text)'
sed 's/^X//' << 'SHAR_EOF' > 't/in/in.1' &&
XFrom aks Fri 22 May 92
To: aks(_at_)hub
Subject: test:aa
Date: Fri, 22 May 92 16:11:08 +22318216
XFrom: Joe bloe <root(_at_)anywhere>
X
This is a test- - ignore.
SHAR_EOF
$TOUCH -am 1229173595 't/in/in.1' &&
chmod 0644 t/in/in.1 ||
echo 'restore of t/in/in.1 failed'
Wc_c="`wc -c < 't/in/in.1'`"
test 151 -eq "$Wc_c" ||
echo 't/in/in.1: original size 151, current size' "$Wc_c"
fi
# ============= t/in/in.2 ==============
if test -f 't/in/in.2' -a X"$1" != X"-c"; then
echo 'x - skipping t/in/in.2 (File already exists)'
else
echo 'x - extracting t/in/in.2 (Text)'
sed 's/^X//' << 'SHAR_EOF' > 't/in/in.2' &&
XFrom owner-mailist(_at_)somebody(_dot_)com Sat Aug 19 09:35:50 1995
Received: from mailer1.somebody.com
X by mailer2.somebody.com (1.38.193.5/v3)
X id AA06412; Sat, 19 Aug 1995 09:35:50 -0700
Received: from somebody.com by mailer2.somebody.com; id AA24564
X sendmail 4.1/sun
X Sat, 19 Aug 95 09:35:42 PDT for me(_at_)myhost(_dot_)somebody(_dot_)edu
Received: by somebody.com (940816.SGI.8.6.9/940406.SGI)
X for mailist-members id VAA23954; Fri, 18 Aug 1995 21:02:10 GMT
XFrom: "Foo Bar" <fubar(_at_)myhost(_dot_)somebody(_dot_)com>
Message-Id: <9508181402(_dot_)ZM10974(_at_)myhost(_dot_)somebody(_dot_)com>
Date: Fri, 18 Aug 1995 14:01:59 -0700
To: mailist(_at_)somebody(_dot_)com
Subject: Fancy Mailing List
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Sender: owner-mailist(_at_)somebody(_dot_)com
Precedence: bulk
X
[ This mail is from a mailing list ]
[ actual mail deleted ]
SHAR_EOF
$TOUCH -am 1229174195 't/in/in.2' &&
chmod 0600 t/in/in.2 ||
echo 'restore of t/in/in.2 failed'
Wc_c="`wc -c < 't/in/in.2'`"
test 801 -eq "$Wc_c" ||
echo 't/in/in.2: original size 801, current size' "$Wc_c"
fi
exit 0