Re: should I trust 'free' or 'top'?

[Adam McKenna]

On Fri, Feb 01, 2002 at 12:11:45PM -0800, William Lee Irwin III wrote:
> On Fri, Feb 01, 2002 at 11:24:16AM -0800, Adam McKenna wrote:
> > adam@xpdb:~$ uptime
> >  11:21am  up 42 days, 18:53,  3 users,  load average: 54.72, 21.21, 17.60
> > adam@xpdb:~$ free
> >              total       used       free     shared    buffers     cached
> > Mem:       5528464    5522744       5720          0        476    5349784
> > -/+ buffers/cache:     172484    5355980
> > Swap:      2939804    1302368    1637436
> > As you can see, there are supposedly 5.3 gigs of memory free (not counting
> > memory used for cache).  However, the box is swapping like mad (about 10 megs
> > every 2 seconds according to vmstat) and the load is skyrocketing.
> 
> That 5.3GB is without kernel caches. I see 5.7MB...
> 
> On Fri, Feb 01, 2002 at 11:24:16AM -0800, Adam McKenna wrote:
> > Now top, on the other hand, has a very different idea about the amount of
> > free memory:
> > CPU states:  0.0% user,  0.1% system,  0.1% nice,  0.0% idle
> > Mem:  5528464K av, 5523484K used,   4980K free,      0K shrd,    340K buff
> > Swap: 2939804K av, 1082008K used, 1857796K free                5351892K
> > cached
> 
> They actually agree. The line you're reading with 5.3GB in it subtracts
> kernel caches from the memory in use.
> 
> The fun bit about swapping like mad is because kernel caches are not
> being flushed and shrunk properly in response to growth of the working
> set. In more concrete terms, the kernel is making decisions which prefer
> to keep things like the page cache, the dentry cache, the inode cache,
> and the buffer cache in memory over the working sets of your programs.
> There is some tradeoff: it is probably also not desirable to allow the
> working set to erode kernel caches to the absolute minimum (or at least
> not very easily), but obviously what tradeoffs are happening here are
> suboptimal for your workload (and generally insufficiently adaptive). It
> appears that when the kernel caches are done with you you've got 172MB
> out of 5.5GB of physical memory left for your programs' anonymous memory.
> 
> What kernel/VM are you using?

2.4.6-xfs but we've also seen this with 2.4.14-xfs (xfs 1.0.2 release)

> Could you follow up with /proc/slabinfo and /proc/meminfo?

We've already rebooted the box, next time we are experiencing the problem
I'll send this info.

Meanwhile, is there any way to tune the kernel cache?

--Adam

-- 
Adam McKenna <[e-mail]>   | GPG: 17A4 11F7 5E7E C2E7 08AA
http://flounder.net/publickey.html |      38B0 05D0 8BF7 2C6D 110A