Re: Ext3 meta-data performance

[Kevin Jacobs]

On Thu, 29 May 2003, Nick Piggin wrote:
> Kevin Jacobs wrote:
> >[...]
> >Since these rsync backups are done in addition to traditional daily tape
> >backups, we've taken the system out of production use and opened the door
> >for experimentation.  So, the next logical step was to try a 2.5 kernel. 
> >After some work, I've gotten 2.5.70-mm2 booting and it is _much_ better than
> >the Redhat 2.4 kernels, and the system interactivity is flawless.  However,
> >the speed of creating hard-links is still three and a half times slower than
> >with the old 2.2 kernel.  It now takes ~14 minutes to create the links, and
> >from what I can tell, the bottlenecks is not the CPU or the disk-throughput. 
> >
> Its probably seek bound.
> Provide some more information about your disk/partition setup, and external
> journals, and data= mode. Remember ext3 will generally always have to do
> more work than ext2.

  SCSI ID 1  3ware 7500-8 ATA RAID Controller

     * Array Unit 0  Mirror (RAID 1)  40.01 GB   OK
          + Port 0 WDC WD400BB-00DEA0    40.02 GB   OK
          + Port 1 WDC WD400BB-00DEA0    40.02 GB   OK
     * Array Unit 4  Striped with Parity 64K (RAID 5)  555.84 GB   OK
          + Port 4 IC35L180AVV207-1    185.28 GB   OK
          + Port 5 IC35L180AVV207-1    185.28 GB   OK
          + Port 6 IC35L180AVV207-1    185.28 GB   OK
          + Port 7 IC35L180AVV207-1    185.28 GB   OK

Disk /dev/sda: 40.0 GB, 40019615744 bytes
255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1       261   2096451   83  Linux
/dev/sda2           262      1566  10482412+  83  Linux
/dev/sda3          1567      4570  24129630   83  Linux
/dev/sda4          4571      4865   2369587+   f  Win95 Ext'd (LBA)
/dev/sda5          4571      4589    152586   83  Linux
/dev/sda6          4590      4734   1164681   83  Linux
/dev/sda7          4735      4865   1052226   83  Linux

Disk /dev/sdb: 555.8 GB, 555847581696 bytes
255 heads, 63 sectors/track, 67577 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdb1   *         1     67577 542812221   83  Linux

Unit 0 is /dev/sda and the journal is /dev/sda5. Unit 1 is /dev/sdb and the
backup filesystem is /dev/sdb1. The data= mode is whatever is default,
/dev/sdb1 is mounted noatime.  I've also applied the journal_refile_buffer
patch posted by AKPM yesterday morning.

> If you want to play with the scheduler, try set
> /sys/block/blockdev*/queue/nr_requests = 8192

This killed the entire system -- livelocking it with no disk activity to the
point that I had to hit the reset button.  So does setting nr_requests on
sda and sdb from 128 to 256.  The problems hit before the rsync, during a
'rm -Rf' on a previously copied tree.

> then try
> /sys/block/blockdev*/queue/iosched/antic_expire = 0

This seemed to make no difference.

> Try the above combinations with and without a big TCQ depth. You should
> be able to set them on the fly and see what happens to throughput during
> the operation. Let me know what you see.

I'm not sure how to change TCQ depth on the fly.  Last I knew, it was a
compiled-in parameter.

I have some more time to experiment, so please let me know if there is
anything else you think I should try.

Thanks,
-Kevin