Re: Where's all my memory going?

[Rolf Lear]

Matt Dainty <[e-mail]> writes:
>Hi,
>
>I've fashioned a qmail mail server using an HP NetServer with an HP NetRaid
>4M & 1GB RAM, running 2.4.17 with aacraid, LVM, ext3 and highmem. The box
>has 6x 9GB disks, one for system, one for qmail's queue, and the remaining
>four are RAID5'd with LVM. ext3 is only on the queue disk, ext2 everywhere
>else.
>
....

qmail is very file intensive (which is a good thing ...), and RAID5 is very resource intensive (every write to RAID5 involves a number of reads and a write).

It is quite conceivable that the data volume (throughput) generated by the tests is too large for the throughput of your RAID system. From your mail I understand that the mails are being delivered locally to the RAID5 disk array.

One explaination for your results are that your qmail queue is being filled (by qmail-smtpd) at the rate of the network (presumably 100Mbit or about 10-12MiB/s). This queue is then delivered locally to the RAID5. Files in the queue do not last long (are created and then deleted, and the cache probably never gets flushed to disk ...). Delivered e-mails fill the cache though, and the kernel at some point will begin flushing these cache entries to disk. At some point (and I am guessing this is your 35-40 minute point) all pages in the cache are dirty (i.e. the kernel has not been able to write the cache to disk as fast as it is being filled ...). This will cause the disk to become your bottleneck.

This is based on the assumption that the RAID5 is slower than the network. In my experience, this is often the case. A good test for this would be tools like bonnie++, or tools like vmstat. On a saturated raid array with a cache, it is typical to get 'vmstat 1' output which shows rapid bursts of data writes (bo's), followed by periods of inactivity. A longer vmstat like 'vmstat 10' will probably even out these bursts, and show an 'averaged' throughput of your disks.

It is possible that I am completely off base, but I have been battling similar problems myself recently, and discovered to my horror that RAID5 disk arrays are pathetically slow. Check your disk performance for the bottleneck.

Rolf