Bug report

[Craig Cummings]

Hi there,

I think this qualifies as a bug but let me know if this could be a
configuration or hardware issue.

I've been having problems with memory leaks when I run programs on
large--up to 250MB text files.  (I know this is huge, but that's the 3
billion base human genome for you.)  At first I though it was a Perl
problem but I later found that a completely unrelated C program also
caused memory leaks.  I recently upgraded to the 2.4 kernel, hoping to
solve these problems (see below).  However, the memory leaks are still
happening and this time I know the problem is at a deeper level than my
programs.  Some standard UNIX programs are leaking a lot of memory.  I
would appreciate some advice and ultimately, a fix.  Unfortunately, My
programming skills are not sufficient for tinkering with the kernel
source.  Thank you, in advance for your help.  Details follow.

Regards,

Craig Cummings

Here are the specs for my system:
	Dell Precision XPSt700, Pentium III, 512 MB RAM
	I've recently upgraded from Red Hat 6.2 with the 2.2.14 kernel to
	Red Hat 7, then built the 2.4.2 kernel on my own.

Here's what happens with grep:

Output of free, freshly booted system:

             total       used       free     shared    buffers     cached
Mem:        513616      47516     466100          0       2476      27048
-/+ buffers/cache:      17992     495624
Swap:       128480          0     128480

Output of free after grep 'NT_005289' Data/hs_chr12.fa:

             total       used       free     shared    buffers     cached
Mem:        513616     183548     330068          0       2624     159616
-/+ buffers/cache:      21308     492308
Swap:       128480          0     128480

Output of grep 'NT_005289' Data/hs_chr2.fa:

>gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment

Output of free after this grep:

             total       used       free     shared    buffers     cached
Mem:        513616     424272      89344          0       2860     394232
-/+ buffers/cache:      27180     486436
Swap:       128480          0     128480

Output of grep 'NT_005289' Data/hs_chr2.fa:

>gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment

Output of free after this grep:

             total       used       free     shared    buffers     cached
Mem:        513616     424272      89344          0       2860     394232
-/+ buffers/cache:      27180     486436
Swap:       128480          0     128480

File sizes of the two files grep'ed:

-rw-rw-r--    1 cummings genomics 135744469 Mar 12 22:09 Data/hs_chr12.fa
-rw-rw-r--    1 cummings genomics 240244039 Mar 12 22:24 Data/hs_chr2.fa

Note that these file sizes are equivalent to the amount of memory leaked
when grep is called on that file.

When I grep the same file a second time, very little additional memory is
leaked.

This same phenomenon occurs when I run a different UNIX program, e.g. wc:

Output of wc -l Data/hs_chr3.fa:

2915465 Data/hs_chr3.fa

Output of free:

             total       used       free     shared    buffers     cached
Mem:        513616     511520       2096          0       1252     481020
-/+ buffers/cache:      29248     484368
Swap:       128480          0     128480

Interestingly, after running wc a second time on the same file, it goes
very fast and very little additional memory is leaked:

             total       used       free     shared    buffers     cached
Mem:        513616     510732       2884          0       1204     480948
-/+ buffers/cache:      28580     485036
Swap:       128480         40     128440

-------------------------------------------
Craig Cummings, Ph.D.

Relman Laboratory
Stanford University School of Medicine
Department of Microbiology and Immunology

e-mail: [e-mail]
phone:  [phone]
fax:    [phone]