On June 26, 1999 at 23:59, Jeff Breidenbach wrote:
Upgrading to 2.4.0 caused out of memory errors when adding to a 60,000
document archive. In version 2.3.3, similar operations only required
about 130 meg, or half of available memory. Running out memory is
serious, because the lack of memory effectively stops the kernel from
starting new processes (causing commands like 'ps' to
coredump). This makes the machine unusable until MHonArc finally exits
with return code 137. Smaller archives appear to work correctly.
Here are some runs of MHonArc v2.4.0 and v2.3.3 on my system
(Note, "mha-work" => v2.4.0, and "mha" => v2.3.3):
119 messages (default settings):
--------------------------------
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15711 ehood 0 0 4320 4320 1036 T 0 0.0 3.3 0:06 mha-work
15712 ehood 10 0 4196 4196 1036 T 0 0.0 3.2 0:06 mha
--------------------------------
Ratio: 1.029552
121 messages (default settings):[1]
--------------------------------
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15696 ehood 0 0 4060 4060 1040 T 0 0.0 3.1 0:05 mha-work
15697 ehood 13 0 3920 3920 1028 T 0 0.0 3.0 0:05 mha
--------------------------------
Ratio: 1.0357143
477 messages (default settings):[1]
--------------------------------
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15689 ehood 18 0 5840 5840 1048 T 0 0.0 4.5 0:23 mha-work
15686 ehood 0 0 5640 5640 1036 T 0 0.0 4.3 0:21 mha
--------------------------------
Ratio: 1.035461
2963 messages (default settings):
--------------------------------
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15577 ehood 0 0 16600 16M 1048 T 0 0.0 12.9 2:37 mha-work
15575 ehood 0 0 16360 15M 1036 T 0 0.0 12.7 2:24 mha
--------------------------------
Ratio: 1.0146699
2963 messages (Jeff's rcfile):
--------------------------------
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
15766 ehood 0 0 14788 14M 1052 T 0 0.0 11.5 2:35 mha-work
15769 ehood 9 0 14576 14M 1040 T 0 0.0 11.3 2:32 mha
--------------------------------
Ratio: 1.0145445
--------------------------------
[1] Digest::MD5 got loaded for mha-work, causing an increase in memory
usage to hold the module.
From my observations, the memory increase is small, and as the number
of messages increases, the ratio of v2.4.0/2.3.3 approached 1.0. I
would guess that with 60,000 messages, the extra memory would amount
to ~1-1.5 MBs. It is interesting that your rcfile actually reduced the
total memory usage of both versions from the default settings.
Note, my system configuration is different from yours. I am running
Perl 5.005_03 on RH 5.2 w/2.2.9 kernel. There is a possibility that
Perl 5.004 has some memory leaks exposed by MHonArc v2.4.0. The way to
check is to run both versions of MHonArc on smaller data sets and check
the process sizes. The way I caught the size info for above is that I
created a variant of the "mhonarc" source file to suspend itself after
processing, but before exiting. Then I can use top(1) (or ps(1) can be
used) to get its size.
Here is source of the variant version:
MAIN: {
unshift(@INC, 'lib'); # Should I leave this line in?
require 'mhamain.pl' || die qq/ERROR: Unable to require "mhamain.pl"\n/;
mhonarc::initialize();
mhonarc::process_input();
kill "STOP", $$; # to suspend ourself (SIGSTOP => 23)
}
Make sure that the mhamain.pl required is getting the proper version
you are testing. You can verify by the following:
shell> mhonarc.variant -v
And check what version is printed out. I had to be careful since I
could be grabbing different libraries depending on what my current
working directory is.
--ewh
