NAS: which ZFS configuration for 8 drives

L

luckylinux

I'm planning on upgrading my home NAS. I was wondering which would the best configuration for my DATA zpool. I'm not looking at ultrahigh performance. I'd be satisfied with let's say 300 MB/s read and writes. I'm not looking for >500 MB/s performances, although I wouldn't want less than 200 MB/s :e .

I'm planning on setting up the OS (ROOT pool) on 2x 240 GB mirror SSD. That leaves 8x 3.0 TB or 4.0 TB (still haven't decided yet) HDDs for the DATA pool. Now it seems eight disks isn't the best configuration for any RAID-Z implementations, since:
  • RAID-Z1 recommends 3, 5, 9, ..., (2N+1) drives
  • RAID-Z2 recommends 4, 6, 10, ..., (2N+2) drives
  • RAID-Z3 recommends 5, 7, 11, ..., (2N+3) drives

Therefore I'm wondering about what to do with an eight drives configuration. "Optimal" possibilities are (assuming 4.0 TB disks):
  • 2 x RAID-Z2 (each vdev made of four HDDs) and no spare HDD (usable space: 16.0 TB, worst case redundancy: two HDDs failing)
  • 1 x RAID-Z3 (one vdev made of seven HDDs) + one spare HDD (usable space: 16.0 TB, worst case redundancy: three HDDs failing)
  • 4 x mirror (each mirror made of two HDDs) and no spare HDD (usable space: 16.0 TB , worst case redundancy: one HDD failing)

Obviously there are also others possibilities. Although sub-optimal I wonder how these would perform.
  • 1 x RAID-Z3 (with eight HDDs, usable space: 20.0 TB)
  • 1 x RAID-Z2 (with eight HDDs, usable space: 24.0 TB)

Of course I'm open to accept any other suggestions ;)
 
Depending on how you back-up that amount of data...

I've just had my first RAID-Z1 pool damaged by one disk failure and another disk being trashed by ESXi passthrough, at the same moment. Fortunately, I had a recent ZFS image of the pool. I'd suggest two pools (2x2 mirror). You could use one pool to partially replicate part of it to the second, for really important stuff and use storage space on both for other "binary data". Just create the pools (e.g. one in mirror and one in RAID-Z2) and test their performance using real file transfers (SAMBA, NFS). Don't use dd or mc benchmarks, they do not really reflect the performance of the ZFS layer.
 
wblock@ said:
I was looking at the comparisons of 6-drive RAID-Z2 versus 7-drive RAID-Z3.
Click to expand...
Not much difference IMHO (~ 30 MB/s faster write in RAID-Z2). Therefore 2 x (RAID-Z2 dev made each of four HDDs) may provide very good performances (about twice as one single RAID-Z3 pool).

HarryE said:
Depending on how you back-up that amount of data...
Click to expand...
I only plan on backing up very important data either using rsync or zfs send (still haven't tried it) to a HP microserver N54L (4 HDDs in RAID-Z2 there).

HarryE said:
I'd suggest two pools (2x2 mirror). You could use one pool to partially replicate part of it to the second, for really important stuff and use storage space on both for other "binary data".
Click to expand...
Your idea is very good, although in my case I think it'd be quite complicated to implement since:
  • Very important data (photos, documents, ...) are maybe like ~ 100 GB (just an order of magnitude)
  • Less important data (movies, ISOs, other big files, ...) are maybe like ~ 2 TB (just an order of magnitude)

Therefore half of the pool would never be used if I understood your suggestion correctly. Please correct me if I'm wrong ;)
 
You can use both pools for storing any data, except for the important one you replicate on both. If one pool fails, you loose only half of the unimportant data. Just set the desired mount point. It doesn't matter if the data sits on different pools. But since you have enough space for backup on N54L, you may create a single pool as well.
 
HarryE said:
You can use both pools for storing any data, except for the important one you replicate on both. If one pool fails, you loose only half of the unimportant data.
Click to expand...
And how would you replicate only a part of the data from POOLA to POOLB? How about having to split collections across pools? That's not very practical for when it has to be retrieved in a directory-like structure.
 
Just create different zfs filesystems on Poola
eg. zfs create poola/myimages
zfs create poola/mymovies
Then you can easily snapshot poola/myimages, zfs send the (incremental) snapshot to poolb without having to do the same for poola/mymovies.
 
luckylinux said:
...
Your idea is very good, although in my case I think it'd be quite complicated to implement since:
  • Very important data (photos, documents, ...) are maybe like ~ 100 GB (just an order of magnitude)
  • Less important data (movies, ISOs, other big files, ...) are maybe like ~ 2 TB (just an order of magnitude)

Therefore half of the pool would never be used if I understood your suggestion correctly. Please correct me if I'm wrong ;)
Click to expand...

What I did was that I have ZFS file systems for storage, important data and very important data. The storage is configured with copies=1, importand has copies=2 and very importand has copies=3. That way, even when enough disks die to damage the pool, the amount of redundancy inside the file systems still provides for a way to read these files. You would then only loose unimportant stuff. That way, in my 8-disk storage server, half of the disks can die and I still keep my family photos intact. Shame about the DVDs backed up there, I had some which are now unreadable in the shelf but their backup is good. But these can be replaced, pictures of my son can not.

That way, you have a better flexibility compared to using two complete seperate pools for seperate file systems and pushing the files around by hand.
 
Crivens said:
What I did was that I have ZFS file systems for storage, important data and very important data. The storage is configured with copies=1, importand has copies=2 and very importand has copies=3. That way, even when enough disks die to damage the pool, the amount of redundancy inside the file systems still provides for a way to read these files.
Click to expand...

I like your approach but at the same time I don't fully understand it.
How many pools do you create out of how many vdevs (and of which configuration each, mirror / RAIDZ2)?
 
That way, in my 8-disk storage server, half of the disks can die and I still keep my family photos intact.
Click to expand...

I'm also confused by this setup. How is your pool configured?
If you have something like RAIDZ2, you're entire pool will be faulted after the loss of two disks. Sure, the number of copies might mean that your important data is available somewhere on the remaining disks, but you won't get at it without recovering the blocks manually, which very few people are capable of.

Copies only really protects against checksum or read errors while the pool is still online. It can't provide access to data beyond the redundancy of the pool itself.
 
As far as I know (but maybe this needs correction) you can force a pool to import even in degraded/faulted state. Then, when the number of copies is sufficient, you can read the data (files which are not sufficiently backed are lost). But after looking up the details in the ZFS Administration Guide, I also have doubt about that. So please disregard that untill I (or someone else) knows better.
 
You must log in or register to reply here.
Back
Top