Recommended pool layout with 12 x 3TB disks?

Xenomorph

Xenomorph

I apologize for making a similar post to my last one (http://forums.freebsd.org/showthread.php?t=35699). I had a hardware question regarding ZFS on a bunch of RAID 0 volumes. I can't seem to find the "Edit" button for that post to remove the pool layout part, as it would probably fit better on this forum.

I was wondering about recommended pool layouts. There are 12 disks, 3TB each. Seagate Constellation ES.2, 7200 RPM.

For testing, I have all 12 disks in one big zraid3 setup. It's obviously not a recommended setup. Performance is OK now, but I'm guessing there will be performance/time issues during a resilvering, due to the size of the vdev.

I'm going for size and safety, instead of speed.
Some options:

Code: 
1x12 raidz3, 3 parity = 27TB
     pool
      raidz3-0
          drive1
          drive2
          drive3
          drive4
          drive5
          drive6
          drive7
          drive8
          drive9
          drive10
          drive11
          drive12

2x6 raidz2, 4 parity = 24TB
     pool
      raidz2-0
          drive1
          drive2
          drive3
          drive4
          drive5
          drive6
      raidz2-1
          drive7
          drive8
          drive9
          drive10
          drive11
          drive12

3x4 raidz1, 3 parity = 27TB
     pool
      raidz1-0
          drive1
          drive2
          drive3
          drive4
      raidz1-1
          drive5
          drive6
          drive7
          drive8
      raidz1-2
          drive9
          drive10
          drive11
          drive12

4x3 raidz1, 4 parity = 24TB
     pool
      raidz1-0
          drive1
          drive2
          drive3
      raidz1-1
          drive4
          drive5
          drive6
      raidz1-2
          drive7
          drive8
          drive9
      raidz1-3
          drive10
          drive11
          drive12
 
Considering how log would it take to resilver a 3 TB drive I would consider two options:

- 2 x RAIDZ2 (effectively 2-way RAID60) (You already mentioned it) which will gave You 24 TB of usable space (not counting the 1000 vs 1024 difference on disks).
- 6 x RAID1 (effectively 6-way RAID10) which will gave 18 TB of usable space and is the fastest safe option.
 
If you do not use dedup, you should be fine with 1x raidz2 or 3. Resilver time with 90% used space should complete in 2-3 days(if you can live with that).

I would recommend to fill your drives with random data and test it first though. Then you get to test if everything else is ok too. Cabling, defective drives from factory and other things.
 
Huge RAID-Z/2/3 sets are notorious for being awfully slow to resilver, so combined with the reduced redundancy that is a bigger window of risk. So I personally prefer a multi-vdev setup because this will improve both performance and redundancy at the expense of some disk space which is cheap :)

We use the 2x RAID-Z2 setup with 3 TB drives on our biggest box. Good redundancy (2 drive failures on each vdev), resilvering a drive only taxes one vdev which takes less than a day when the drives+controller are fast enough and not too full.
 
One thing to keep in mind is that, when one drive dies, the others from the same batch would likely die also during the extra stress of the resilver. The "sweet spot" for the extra risk of death-during-resilver over capacity was IIRC somewhere between 500G and one TB.

Also take care not to buy all drives from the same manufacturer or the same batch. I used disks from two seperate manufacturers, and then bought half of those from different suppliers giving me 4 different {drive,batch} sets which would hopefully not all die in the same timeframe. Then build the vdevs from these disks so no two from one set are in one vdev.
 
You must log in or register to reply here.
Back
Top