I'm in the process of setting up RAID15 and RAID51. Right now, this is
for performance testing, but if operational characteristics are good,
it will go into production. The hardware platform is a dual-processor
P-III box (1Ghz processors), an ASUS CUR-DLS board, 2GB RAM, and two
3ware 8-channel IDE controllers (sitting on seperate PCI busses).
There are 16 disk drives, Maxtor 80GB beasts. The kernel is 2.4.17-rc2,
with the MD code compiled in (no modules). I use the raidtools2 package
of current Debian woody. As a testing platform, the machine does not
do anything besides the things described here:
In order to compare RAID15 to RAID51, I split the drives into two equal-sized
partitions, sd[a-p]2 for the RAID15, and sd[a-p]3 for the RAID51 tests.
The first layer was created yesterday: 8 mirror pairs, each pair with one
drive on each 3ware controller, rebuilt in parallel at a rate of about
7.5MB/s per mirror pair, 60MB/s aggregate. Next, two RAID5 sets were
created in parallel, each spanning the "3" partition of the 8 drives
of one, and the other, controller. This initial rebuild progressed
at about 12MB/s/set, 24MB/s aggregate. I waited until all this layer 1
rebuild was done.
So far, everything is well. I/O characteristics for sequential read and
write are OK on both variants. Sequential read from the RAID5 sets gives
me a whopping 150MB/s transfer speed. Great.
Then I started the second layer, making a RAID5 set out of the 8 mirror pairs
out of the "2" partitions, and making a mirror pair out of the two RAID5 sets
on the "3" partition. Technically, this seems to work, but the initial rebuild
runs extremely slow, compared to the above rates: the RAID15 rebuild goes
at about 380kB/s, and the RAID51 build at slightly over 100kB/s. I will report
on operational characteristics, once these rebuilds are through (in about
30 days, for the RAID51...)
Anybody got any explanation, hints, or even convincing discouragement?
best regards
Patrick
Please Cc: me on replies, I'm not subscribed to linux-kernel.
Here is the /etc/raidtab I am using. md[0-7] are the layer 1 mirror pairs,
md1[0-1] are the layer 1 RAID5 arrays, md8 is the RAID15 thing, and md9
is the RAID51 thing.
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sda2
raid-disk 0
device /dev/sdi2
raid-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdb2
raid-disk 0
device /dev/sdj2
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdc2
raid-disk 0
device /dev/sdk2
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdd2
raid-disk 0
device /dev/sdl2
raid-disk 1
raiddev /dev/md4
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sde2
raid-disk 0
device /dev/sdm2
raid-disk 1
raiddev /dev/md5
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdf2
raid-disk 0
device /dev/sdn2
raid-disk 1
raiddev /dev/md6
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdg2
raid-disk 0
device /dev/sdo2
raid-disk 1
raiddev /dev/md7
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sdh2
raid-disk 0
device /dev/sdp2
raid-disk 1
raiddev /dev/md8
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 128
device /dev/md0
raid-disk 0
device /dev/md1
raid-disk 1
device /dev/md2
raid-disk 2
device /dev/md3
raid-disk 3
device /dev/md4
raid-disk 4
device /dev/md5
raid-disk 5
device /dev/md6
raid-disk 6
device /dev/md7
raid-disk 7
raiddev /dev/md9
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/md10
raid-disk 0
device /dev/md11
raid-disk 1
raiddev /dev/md10
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 32
device /dev/sda3
raid-disk 0
device /dev/sdb3
raid-disk 1
device /dev/sdc3
raid-disk 2
device /dev/sdd3
raid-disk 3
device /dev/sde3
raid-disk 4
device /dev/sdf3
raid-disk 5
device /dev/sdg3
raid-disk 6
device /dev/sdh3
raid-disk 7
raiddev /dev/md11
raid-level 5
nr-raid-disks 8
persistent-superblock 1
chunk-size 32
device /dev/sdi3
raid-disk 0
device /dev/sdj3
raid-disk 1
device /dev/sdk3
raid-disk 2
device /dev/sdl3
raid-disk 3
device /dev/sdm3
raid-disk 4
device /dev/sdn3
raid-disk 5
device /dev/sdo3
raid-disk 6
device /dev/sdp3
raid-disk 7
Finally, here's a current snapshot of /proc/mdstat:
Personalities : [raid0] [raid1] [raid5]
read_ahead 1024 sectors
md8 : active raid5 md7[7] md6[6] md5[5] md4[4] md3[3] md2[2] md1[1] md0[0]
272702080 blocks level 5, 128k chunk, algorithm 0 [8/8] [UUUUUUUU]
[========>............] resync = 43.9% (17120184/38957440) finish=909.8min speed=396K/sec
md9 : active raid1 md11[1] md10[0]
272814912 blocks [2/2] [UU]
[>....................] resync = 1.7% (4651736/272814912) finish=39435.7min speed=111K/sec
md11 : active raid5 sdp3[7] sdo3[6] sdn3[5] sdm3[4] sdl3[3] sdk3[2] sdj3[1] sdi3[0]
272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
md10 : active raid5 sdh3[7] sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
md0 : active raid1 sdi2[1] sda2[0]
38957504 blocks [2/2] [UU]
md1 : active raid1 sdj2[1] sdb2[0]
38957504 blocks [2/2] [UU]
md2 : active raid1 sdk2[1] sdc2[0]
38957504 blocks [2/2] [UU]
md3 : active raid1 sdl2[1] sdd2[0]
38957504 blocks [2/2] [UU]
md4 : active raid1 sdm2[1] sde2[0]
38957504 blocks [2/2] [UU]
md5 : active raid1 sdn2[1] sdf2[0]
38957504 blocks [2/2] [UU]
md6 : active raid1 sdo2[1] sdg2[0]
38957504 blocks [2/2] [UU]
md7 : active raid1 sdp2[1] sdh2[0]
38957504 blocks [2/2] [UU]
unused devices: <none>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/