>From phk(_at_)critter(_dot_)freebsd(_dot_)dk Thu Sep 18 14:32:51 1997 Return-Path: Delivered-To: m(_dot_)koster(_at_)greenhills(_dot_)co(_dot_)uk Received: (qmail 7055 invoked from network); 18 Sep 1997 14:32:50 -0000 Received: from punt-2b.mail.demon.net (HELO relay-6.mail.demon.net) (194.217.242.6) by mindy.greenhills.co.uk with SMTP; 18 Sep 1997 14:32:50 -0000 Received: from punt-2.mail.demon.net by mailstore for m(_dot_)koster(_at_)greenhills(_dot_)co(_dot_)uk id 874591153:12:05626:2; Thu, 18 Sep 97 14:59:13 BST Received: from mail.webcrawler.com ([204.62.245.201]) by punt-2.mail.demon.net id aa0526954; 18 Sep 97 14:58 BST Received: (qmail 11312 invoked by uid 982); 18 Sep 1997 13:54:49 -0000 Delivered-To: mak(_at_)webcrawler(_dot_)com Received: (qmail 11309 invoked from network); 18 Sep 1997 13:54:47 -0000 Received: from critter.freebsd.dk (195.8.129.26) by mail.webcrawler.com with SMTP; 18 Sep 1997 13:54:47 -0000 Received: from critter.freebsd.dk (localhost.cybercity.dk [127.0.0.1]) by critter.freebsd.dk (8.8.7/8.8.7) with ESMTP id PAA03511; Thu, 18 Sep 1997 15:57:51 +0200 (CEST) To: Martijn Koster cc: Andre Albsmeier Subject: Re: procmail dies, why? In-reply-to: Your message of "Thu, 18 Sep 1997 14:25:43 BST." <19970918142543(_dot_)12647(_at_)webcrawler(_dot_)com> Date: Thu, 18 Sep 1997 15:57:51 +0200 Message-ID: <3509(_dot_)874591071(_at_)critter(_dot_)freebsd(_dot_)dk> From: Poul-Henning Kamp Status: RO X-Status: A Content-Length: 5001 Lines: 180 There isn't much that can be done in this case, it's a singular stupid way of allocating memory. I would need to do page flipping AND be very lucky to be able to make this work decently. A far better way to do the job is to: mul = 16 size = 64K; malloc (size) while (not enough) /* don't mess around */ size *= mul realloc(size) if (mul > 2) mul <<= 1; Don't worry about overallocation, it's practically free until you actually access the memory. even doing malloc (32M) right away would be cheaper than the current stuff. But even smarter would be to pipe the stuff into a temporary file and them mmap it... Procmail is stupid, and should be fixed. Poul-Henning In message <19970918142543(_dot_)12647(_at_)webcrawler(_dot_)com>, Martijn Koster writes: > >[phk: we briefly discussed this last week. Can I once again ask for help?] > > >On Thu, Sep 18, 1997 at 01:27:24PM +0200, Andre Albsmeier wrote: > >> I have here 8 2.2-STABLE machines and all behave equal. realloc or >> procmail screws up the machine although with the new malloc.c >> the system doesn't crash any more. > >You are right; malloc 1.32 improves the situation, but something is >still weird. > >Poul -- can you reproduce this? Any ideas what's up? > >When I run rea.c with 100M argument on -stable (with malloc.c-1.32) on >a 64M machine (132.96-MHz 586-class CPU), I get to 21M, when the >machine has run out of swap (138M). One -current on a 64M machine >(199.43-MHz 686-class CPU) I get to 25M, by then the machine has run >out of swap (138M). This process takes some 12 minutes (!). > >This code runs fine up to 100M on Irix 6.2 (Indy, 96M, 2 secs) and >Solaris 2.5 (Sparc5, 64M, 10 secs). This doesn't make FreeBSD look >good :-( > >Below is all the information I think may be relevant for reproduction. > >-- Martijn > >mak(_at_)pitch$ cat rea.c >#include >#include >#include > >int >main(int argc, char* argv[]) >{ > size_t size = 0; > char *p = NULL; > int max = 8404992; > > if (argc > 2) { > fprintf(stderr, "Usage: %s maxmem\n", argv[0]); > exit(-1); > } > if (argc > 1) > max = atoi(argv[1]); > > while(1) { > > size += 16384; > if (size > max) > break; > > fprintf(stderr, "realloc(%u)\n", size); > if ((p = realloc(p, size)) == NULL) { > fprintf(stderr, "out of memory\n"); > exit(-1); > } > } > > { > int c; > printf("done. press return to quit\n"); > read(0, &c, 1); > } >} >mak(_at_)pitch$ uname -a >FreeBSD pitch.webcrawler.com 2.2-STABLE FreeBSD 2.2-STABLE #0: Tue Sep 9 16:0 >4:50 PDT 1997 mak(_at_)pitch(_dot_)webcrawler(_dot_)com:/usr/src/sys/compile/PITCH i386 >mak(_at_)pitch$ dmesg | grep 'real memory' >real memory = 67108864 (65536K bytes) >mak(_at_)pitch$ grep '\$Id' /usr/src/lib/libc/stdlib/malloc.c > * $Id: malloc.c,v 1.32 1997/08/31 05:59:39 phk Exp $ >mak(_at_)pitch$ ls -l /etc/malloc.conf >ls: /etc/malloc.conf: No such file or directory >mak(_at_)pitch$ cc -c /usr/src/lib/libc/stdlib/malloc.c >mak(_at_)pitch$ cc -c rea.c >mak(_at_)pitch$ cc -o rea rea.o malloc.o >mak(_at_)pitch$ sudo su >Password: ># bash >bash-2.00# for i in -c -d -f -l -m -n -p -s -t -u -v >> do >> echo $i >> ulimit $i unlimited >> done >-c >-d >-f >-l >-m >-p >bash: ulimit: cannot modify limit: Invalid argument >-s >-t >-u >-v >bash: ulimit: cannot modify limit: Invalid argument >bash-2.00# ulimit -a >core file size (blocks) unlimited >data seg size (kbytes) 131072 >file size (blocks) unlimited >max locked memory (kbytes) unlimited >max memory size (kbytes) unlimited >open files 360 >pipe size (512 bytes) 1 >stack size (kbytes) 65536 >cpu time (seconds) unlimited >max user processes 179 >virtual memory (kbytes) 196608 >bash-2.00# ./rea 104857600 ># ... many lines of output >realloc(21823488) >out of memory >bash-2.00# > ># At the time, top showed: > > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND >13061 root -18 0 42756K 34196K swread 13:10 21.51% 21.51% rea >11882 mak 10 0 776K 436K wait 0:23 0.84% 0.84% bash >last pid: 16331; load averages: 1.50, 1.70, 1.36 22:32:54 >32 processes: 1 running, 31 sleeping > >Mem: 26M Active, 9660K Inact, 19M Wired, 5876K Cache, 7616K Buf, 532K Free >Swap: 138M Total, 137M Used, 1684K Free, 99% Inuse > > > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND >13061 root -18 0 42756K 35776K swread 13:10 20.56% 20.56% rea >11882 mak 10 0 776K 436K wait 0:23 0.88% 0.88% bash >head: Cannot allocate memory >kvm_open: Cannot allocate memory >kvm_open: Cannot allocate memory >head: Cannot allocate memory >head: Cannot allocate memory >kvm_open: Cannot allocate memory >kvm_open: Cannot allocate memory >head: Cannot allocate memory > -- Poul-Henning Kamp FreeBSD coreteam member phk(_at_)FreeBSD(_dot_)ORG "Real hackers run -current on their laptop."