Question about ZFS/Raidz

B

Blu

I have a question about ZFS/Raidz set up that I plan on using for my file server (making it a NAS solution) but my questions are, I plan on having 20 2TB drives, should I opt for Raidz2 for better redundancy? Also I hear ZFS/Raidz eats alot of ram, how much ram would you say? I'm willing to put 16GB in if it will help speed up the overall speed of my file server.

What are the best hard drives to use for Raidz? I was looking at the 2TB greens since I got a ton of them and they've never done me wrong but a friend suggested hitachi drives but I just read that WD bought hitachi or are going to in a few days/weeks here. Will this also work? If I get a 16 port SAS/Sata card and then connect my other drives to my motherboard will that work with software raid? Also does Raidz need a hard drive to install to? I was thinking of buying a cheap SSD for about $30-50 to be the main OS drive on it.

Another question, if my parity drive fails, will raidz tell me what drive failed based on serial number or something? So I don't have to go through 20 drives to see what failed, also will it rebuild the parity drive or any drive that fails on me if I replace it with a new one? I'm basically looking for a massive storage system for my home network for HD movies and HD tv shows. I'm going to post my current build for my file system and you guys tell me what you think.

  • Antec twelve hundred case
  • 4x http://www.newegg.com/Product/Product.aspx?Item=N82E16817994077&cm_re=5_in_3-_-17-994-077-_-Product
  • Mobo: 880GMA (Free from a old build I was going to do on my htpc but I needed more pci slots
  • Ram: 16GB Corsair (Like I said Ive heard alot of people say raidz and zfs eat ALOT of ram)
  • Power supply (to be determined) $50-$10o
  • 880GMA motherboard (Free!)
  • 24 port sas card - $400 with shipping
  • 12 4 pin molex connectors for the power for the drives.
  • 30GB SSD $?????? Might buy this new or second hand, don't know how much it will cost so far.
  • H50 Cooler for the heck of it :P -$70 (will get on sale for about 50)
  • AMD Sempron 145 Processor 2.8GHz NCIX.com - Buy AMD Sempron 145 Processor 2.8GHZ Socket AM3 1MB L2 Cache 45W Retail Box - AMD - SDX145HBGMBOX - in Canada - $50
  • UPS in case of a power outage, NCIX.com - Buy Cyberpower CP1500AVRLCD 1500VA 900W UPS 8 Outlet 1500 Joules AVR LCD RJ11/RJ45 & 1 USB Serial Black - CyberPower - CP1500AVRLCD - in Canada - $160 on sale (or I'll get a cheaper one but this could power my file server, my hdtv and my moms computer :)
  • 2 120mm fans.

TOTAL: $3420

Hope prices go down a bit by the time I order though, should I swap the sempron out for a cheap quad core cpu as well? Or get one of the duals that are known for being able to unlock 4 cores (if someone can tell me the name of that cpu again I would love that)

So what do you guys think? I probably won't build this for a few months still as I save for it. But yea let me know what you would change and such. Thanks!
 
Most, if not all, of your questions have been asked many times already, and the answers can be found via the search function. :)

For a home server, the WD Green drives are okay. You really can't beat the price. However, they are not ideal for many reasons, all of which are detailed in many, many, many threads here. If you can, avoid them. If you can't get past the super-low price ... don't expect great performance. :)

Don't skimp on the CPU. A single-core Sempron will not be able to keep up with multiple raidz vdevs, especially if you enable compression and/or dedupe. Get a quad-core, or more if possible.

Don't skimp on RAM. Get the most you can afford. For a server with 20 drives, 8 GB should be the minimum. If you can afford the 16 GB, then get it. ZFS will use it all for read cache, limiting the I/O to the disks, and increasing overall system performance.

Don't skimp on the power supply. You'll need a beefy one to power all 20 drives at once. Our 24-drive behemoths have 1300W redundant PSUs. Under 1000W probably won't cut it.

As to the pool setup, you'll want to go with multiple raidz2 vdevs. 2 TB drives will take several days to resilver (rebuild). If you lose a second drive while the first is rebuilding, using raidz1 ... your data is gone!! With 20 drives, you can do 3 raidz2 vdevs using 6-drives each (8 TB usable space per vdev), and have 2 ports left over for spares, or for the OS drives, or whatever.

If you use glabel(8) to label the drives according to where they are (disk01, disk02; or slot01, slot02, etc), and then build the pool using the /dev/label/<name> devices, you'll be able to find failed drives right away.
 
So your saying for every 8 drives to make a vdev and raidz2 it? So overall out of all the drives I would be loosing six to parity? The vdevs you suggested only add up to 18 drives when I'm looking at doing 20, would it be ok to do it this way?

10 drives per vdev and 2 for raidz for parity and x2 this method so overall I would have 32TB of data. Also I'll take your word on upgrading the cpu, I'm getting so many mixed reviews here, some people are saying 16gb ram is way overkill and that 2gb would be good enough, others disagree and such. I was also planning on trying unraid out but a friend told me that NFS/Raidz was so much better and safer to protect your data.

Is this SAS/Sata card good for adding the drives?
http://www.ncix.com/products/?sku=37999&vpn=AOC-SASLP-MV8&manufacture=SuperMicro
I plan on getting two of them since my other choice is out of stock and EOL'd and I fear this one might be EOL'd soon as well, so I might just pick up three or four of them (I found a store online with only 4 left in stock) and use two has back ups incase anything bad happens to the current card. As for the cpu, should I risk it and go with a dual core that has a chance of unlocking to a quad? (currently have one sitting waiting to be put in my HTPC and I'm hoping it's a quad) or just get the sure thing, or possibly get a 6 core cpu for it? You really think I'll need a 1k psu for 20 drives? I've heard drives only use about 10watts of power in full usage mode, but then again I'm getting mixed reviews on this as well like crazy, also would you suggest Hitachi or samsung drives for this build? And is the UPS a good idea so if the power goes out when it's writing data? I would be able to send the command to the pc to shut down after say 5 minutes of no power and save my data correct?
 
Hi,

I think people are generally exaggerating about ZFS and RAM, if you have at least 4GB then this is quite sufficient that ZFS will run normally under most circumstances (ie for a file server should be fine). Additional RAM can be added to provide more RAM for the ARC cache, this is simply to provide more read cache for better performance.
I'd agree with the comments from Phoenix, possibly with the exception of CPU. I have old P4 era Xeons running ZFS fine with compression. So you don't necessarily need anything amazingly powerful, unless you plan on using GZIP compression (lzjb is the default and is CPU light) or possibly dedup (I have no experience using dedup).

thanks Andy.
 
Blu said:
So your saying for every 8 drives to make a vdev and raidz2 it? So overall out of all the drives I would be loosing six to parity? The vdevs you suggested only add up to 18 drives when I'm looking at doing 20, would it be ok to do it this way?

10 drives per vdev and 2 for raidz for parity and x2 this method so overall I would have 32TB of data. Also I'll take your word on upgrading the cpu, I'm getting so many mixed reviews here, some people are saying 16gb ram is way overkill and that 2gb would be good enough, others disagree and such. I was also planning on trying unraid out but a friend told me that NFS/Raidz was so much better and safer to protect your data.
Click to expand...

It all depends on which is more important to you: speed or storage space.

If you want speed, you need multiple small vdevs. As in, 3x 8-disk raidz2.

If you want storage space, you want fewer, large vdevs. As in, 2x 10-disk raidz2.

The more drives you have in a raidz vdev, the longer it takes to resilver a dead drive, and the higher your chances of losing a second drive.

When I did my first ZFS box, I created a single 24-drive raidz2 vdev. Created fine, worked fine, copied data into without issues, everything looked good. Until the first drive died. Spent over two weeks working on it, trying to get that drive to resilver. Never finished. Ended up having to rebuild the box using 3x 8-drive raidz2 vdevs.

The newest ZFS box I'm building, I'm using 4x 6-drive raidz2 to get the better performance. ZFS stripes writes across all the vdevs (basically RAID0), so the more vdevs you have ... the better your overall performance.

Is this SAS/Sata card good for adding the drives?
http://www.ncix.com/products/?sku=37999&vpn=AOC-SASLP-MV8&manufacture=SuperMicro
Click to expand...

I would avoid the Marvell chipsets, I haven't seen any positive reports from ZFS users. Look into the AOC-USAS-L8i cards instead. These use the LSI1068 chipset, fully supported by the mpt(4) drive in FreeBSD. They're inexpensive, but well performing. (Plus, it's what we're using in our newest ZFS box instead of 3Ware 9650 controllers).

As for the cpu, should I risk it and go with a dual core that has a chance of unlocking to a quad?
Click to expand...

For a home server, sure, why not? ;)

You really think I'll need a 1k psu for 20 drives? I've heard drives only use about 10 watts of power in full usage mode, but then again I'm getting mixed reviews on this as well like crazy,
Click to expand...

It's not only the "full load" you need to worry about, but all the "peak power" used to spin up all the drives. Do you really want to risk brown-outs internally? *NEVER* skimp on power. Dirty power, low voltage, over voltage, spikes, etc will do the same damage to low-end hardware as it will to expensive "enterprise" hardware, it doesn't discriminate. :)

If you are using Green drives, you'll be hitting the "peak" power load a lot, as the drives go to sleep, spin down, wake up, spin up, etc.

also would you suggest Hitachi or samsung drives for this build?
Click to expand...

Drive preference is a personal thing. :) Everyone has their favourites. Currently, we're using Western Digital RE3 Black drives (500 GB and 1 TB), and Seagate 7200.11 1 TB, and Seagate 7200.12 1.5 TB drives with great success.

And is the UPS a good idea so if the power goes out when it's writing data? I would be able to send the command to the pc to shut down after say 5 minutes of no power and save my data correct?
Click to expand...

Definitely desirable if you can afford it. ZFS can tolerate loss of power fairly well, depending on how large the caches are on the drives, and whether or not they honour the "cache flush" command. If you can do orderly shutdowns, all the better. :)

My home ZFS server is a crappy P4 box that locks up several times a week due to heat issues. In the almost 2 years it's been running at home, I haven't lost a single file or had a single block of permanent corruption (as noted by zpool()).
 
AndyUKG said:
I think people are generally exaggerating about ZFS and RAM, if you have at least 4GB then this is quite sufficient that ZFS will run normally under most circumstances (ie for a file server should be fine).
Click to expand...

While it's true you can tune a ZFS system to work with less RAM (my home NFS server only has 2 GB, for example), the more RAM you can put into a ZFS box, the better things will run.

Especially in a system with 20 drives, and tens of TB of disk space. The more of that you can cache in the ARC, the better. :)

And, remember, an L2ARC (cache vdev) is not a replacement for RAM, as you need RAM To track the contents of the L2ARC. :)

And, enabling dedupe requires storing the dedupe table (DDT) in ARC/L2ARC, so even more RAM is needed.

If you tune your system, and don't use all the extra features, you can get away with < 8 GB of RAM. But if you want to use all the extra, fancy features of ZFS, stuff the box as full of RAM as your budget can support. :)
 
Sorry for the double post, I can't edit my previous post there.
I'm starting to think of going with two vdevs of 10 drives each and then raidz2 to have two parity drives (that's correct right?) Also should I get a small SSD for the OS to run ZFS/Raidz over the network? Thanks again guys for the help, you've done alot to convince me what to choose, I was going to go unraid but it seems like it's JBOD so I would need to say, make a movies folder on each drive to make one massive drive and thats just a headache.
 
I forgot to ask as well, whats the best sata port card for ZFS/Raidz since that one i showed earlier has the marvel chip and you guys say its not a good idea?
 
Blu said:
I'm starting to think of going with two vdevs of 10 drives each and then raidz2 to have two parity drives (that's correct right?)
Click to expand...

Each raidz2 vdev would have double-parity redundancy, meaning you could lose 2 drives in each vdev (4 drives in total), before losing all the data in the pool. The parity data is intermixed with the normal data; it's not 2 separate parity drives.

Also should I get a small SSD for the OS to run ZFS/Raidz over the network?
Click to expand...

If you have room in the case, then I'd go for an SSD (or two) for the OS. You only need about 10 GB for the OS and any apps you want to install. Then you can use the rest of the SSD for swap and L2ARC space.
 
Say I lost three drives in one of the 10 drive arrays, then I would loose all that data correct? What OS would I need to install for this as I just want it to act as a media/file server on my home network. With raidz is it easy to tell what hard drive failed or no? Like would I have to go through every single drive to see what one failed?

HPA is a option on gigabyte motherboards that really seems to mess up raid arrays from what I've read.
 
Also can you guys suggest me some SAS/Sata port cards that work well with ZFS/Raidz? Hopefully a place that accepts paypal and ships to Canada.
 
And one more thing, is it possible to say start with 10 drives and make two vdevs with raidz2 and then in a couple weeks add 10 more drives (5 to each vdev) without needing to rebuild the whole vdev? How long do you think it would take the parity drives to rebuild a failed drive as well?
 
Blu said:
And one more thing, is it possible to say start with 10 drives and make two vdevs with raidz2 and then in a couple weeks add 10 more drives (5 to each vdev) without needing to rebuild the whole vdev? How long do you think it would take the parity drives to rebuild a failed drive as well?
Click to expand...

You can't add disks to a vdev, but you can add new vdevs to an existing pool.

The time to rebuild depends very much on number and size of disks, and the level of filesystem activity. Replacing a 500Gb disk in a 3-disk raidz with no other read/write activity takes about 2 hours. I guess that would scale pretty much linearly with disk/vdev size, but with any amount of ongoing IO rebuildtime might actually go through the roof.
 
Blu said:
And one more thing, is it possible to say start with 10 drives and make two vdevs with raidz2 and then in a couple weeks add 10 more drives (5 to each vdev) without needing to rebuild the whole vdev? How long do you think it would take the parity drives to rebuild a failed drive as well?
Click to expand...

As mentioned by Jalla, the way you need to do with this with ZFS if you want to start your config with 10 drives then got to 20 is first create 1 RAIDZ2 of 10 drives, then when you get the other 10 drives create another. If the two RAIDZ2 devices are in the same zpool, the extra space is automatically available to any existing file systems (ie all file systems will grow by the space added) which I guess is your requirement.

Re which SAS/SATA card, I am using cards based on Sil3124 (eSATA) with port multipliers disk shelves.

thanks Andy.
 
To repeat again, on the zpool expansion... ZFS is designed for the typical "always grow" installation. You can start with single vdev of say, 5 drives. The vdev can be raidz1 or raidz2 (one or two "wasted" drives for redundancy). Then you could add another vdev of 5 drives, then another etc.

In theory, you could add different type of vdev to the pool. Single drive (no redundancy), mirror or raidz1, raidz2. This is rarely recommended but in principle possible.
Again, remember that if you have single non-redundant vdev, your entire zpool is non-redundant!

When you add new vdevs, there is no resilver, nothing is rebuilt or recalculated. It happens instantly. Any old data remains where it is (on old, existing vdevs), any new data is spread all over.

Currently, it is not possible to remove vdevs from an zpool, so plan wisely.
 
Blu said:
When I check the card on super micros site it lists only a few compatible motherboards, it says "UIO Motherboards" should I just disregard that? It should work fine on a normal mobo right.
Click to expand...

UIO cards are normal PCIe (PCI-Express) cards. The only difference is that SuperMicro ships them with the backplane connector on the "wrong" side of the card. That way, they can sell "UIO" motherboards for a premium, that will work with the UIO cards. However, if you just remove the backplane from the card, it works perfectly well in any motherboard with PCIe slots.

There are even some places online where you can order normal connectors for these cards, if you really want to screw them into the case.
 
Blu said:
Say I lost three drives in one of the 10 drive arrays, then I would loose all that data correct?
Click to expand...

You would lose all data in the pool.

What OS would I need to install for this as I just want it to act as a media/file server on my home network.
Click to expand...

:) Obviously, FreeBSD. :)

With raidz is it easy to tell what hard drive failed or no? Like would I have to go through every single drive to see what one failed?
Click to expand...

Are you not reading what I write? :)

Use glabel(8) to label the disks with names that correspond to where they are located in the box. ie "disk01", "disk02", etc. Or even "slot1", "bay03", etc.

Then you create the ZFS pool using the /dev/label/<name> devices. When a drive fails, it will be listed in the output of # zpool status making it very easy to find and replace.
 
Blu said:
And one more thing, is it possible to say start with 10 drives and make two vdevs with raidz2 and then in a couple weeks add 10 more drives (5 to each vdev) without needing to rebuild the whole vdev? How long do you think it would take the parity drives to rebuild a failed drive as well?
Click to expand...

Le sigh. :(

Now would be a good time to start reading through the man pages for zpool() and zfs(), along with the ZFS Best Practises guide, and maybe some of the historical blog posts about ZFS. This is very basic information that is covered very well in there.

Searching the forums for threads on ZFS would also be helpful, as this exact question is answered at least a dozen times already.
 
AndyUKG said:
As mentioned by Jalla, the way you need to do with this with ZFS if you want to start your config with 10 drives then got to 20 is first create 1 RAIDZ2 of 10 drives, then when you get the other 10 drives create another. If the two RAIDZ2 devices are in the same zpool, the extra space is automatically available to any existing file systems (ie all file systems will grow by the space added) which I guess is your requirement.

Re which SAS/SATA card, I am using cards based on Sil3124 (eSATA) with port multipliers disk shelves.

thanks Andy.
Click to expand...

Ok that sounds perfect, so I would be able to add 10 more drives as raidz2 to the zpool and have it show up as two networked drives worth of 16TB Space each (In windows under 'my computer -networked drives-') Right?
 
Another odd question, does Raidz show me the drives serial number to make it easier to keep track of the drives? Or am I going to have to say add the hard drives one by one to keep track of which is which and label them in free BSD as I do this? I know some windows programs will show you the drives serial # and I figured this would be easy to keep track of them. To write down the serial on a piece of paper and tape it to the enclosure the drive is in and such to make it easier to know what drive is what when I use glabel to mark them in the zpool. Also you say it will be listed in the command "zpool status" don't you mean it won't appear there? or will that just show me what drives are missing from the zpool. Just want to make sure I got all this right cause I'm hoping to start ordering some parts soon.
 
Also what would you guys say my data would be safer on? A unraid system or a Raidz2 System?
 
You must log in or register to reply here.
Back
Top