I've gone through the mail archives but can't find a solution to my problem.
I've been using mhonarc to process emails that are being generated by nightly
builds of products within the company I work for.
This has been fine until recently when the developers enabled more verbose
logging of the builds within their tool.
The mail messages that are being processed are stored in maildir format and
they are sent as an email with a html attachment that is to be processed.
Looking at the raw format of the emails the html is encoded using base64.
The MS Visual Build Professional build tool is now producing ridiculously large
HTML reports up to 34 MB in size (typical MS tool I guess).
Unfortunately I don't have control over that.
However the system I am running mhonarc on has 512MB of physical RAM, and
1024MB of swap allocated, so in theory should be able to process these files
The command I am using to process the folder is as follows:
/usr/bin/mhonarc -add -quiet -savemem -rcfile /usr/local/etc/builds.mrc
-outdir /var/www/build-reports /var/local/sharedfolders/.build/cur/
I was previously not using the -savemem parameter, but came across it on the
mailing lists and gave it a shot but it still fails.
Unfortunately when mhonarc chews up all the memory, the kernel out-of-memory
killer steps in and usually clobbers a mysql database running on the system
that contains performance monitoring data.
The version of mhonarc I am using is 2.6.16 and my builds.mrc file referred to
in the above command has the following contents (hopefully it doesn't wrap):
<!-- MHonArc Resource File -->
<!-- Run as:
mhonarc -add -quiet -savemem -rcfile /usr/local/etc/builds.mrc -outdir
<!-- Set Options. -->
<!-- Since we will be pointing at Maildir directories, set the mhpattern
<!-- Expire messages after 30.5 days -->
<!-- Leave text/plain attachements as attachments and don't expand them
m2h_text_html::filter; allowcomments allowscript
<!-- Format used for day group heading. -->
%B %d, %Y
<!-- Redefine LISTBEGIN since a table will be used for index listing.
<table border=1 frame="box" rules="groups" width=100%>
<!-- DAYBEGIN defines the markup to be printed when a new day group is
<!-- DAYEND defines the markup to be printed when a day group ends. No
markup is needed in this case, so we leave it blank. -->
<!-- Define LITEMPLATE to display the time of day the message was sent,
message subject, author, and any annotation for the message.
<!-- Define LISTEND to close table -->
The current version of perl is 5.8.8.
The config above was set in conjunction with the developers to get it into a
format that they are happy with.
The build folder has 1 weeks work of message which at the moment stands at 86
Most message have been up to around 2MB max in the past, but as of yesterday,
it's now produced two massive 30MB messages, and a few smaller ones again.
When using top to monitor, mhonarc gets up to around 407MB or resident RAM size
I can't remember the virtual memory size it allocated, but it was over 2GB.
These figures are pretty excessive for processing at most, a 34MB file.
Any solution on how to solve the problem would be very helpful.