ZFS-specific advice for a homebuilt NAS

pineconez · Apr 24, 2012

Cheers.

I'm in the process of assembling a NAS/file server for home use (i.e. backups, video streaming, the usual stuff) and the folks over at Tom's Hardware (here's my thread) have been very helpful so far.

I really want to use ZFS for this (which platform I don't know yet), and there are some points I need clarification on.

First: How's the performance difference between RAID-Z2 and a pool of mirrored vdevs? I know the latter will have significantly better performance, but will it matter in a home environment (considering I'll have to at most saturate 1Gig Ethernet). On the other hand, I'd like to use vdevs for simplicity: anytime I want to expand the server, I just need to put two drives in and add them.

Second: What happens with the storage drives if the OS drive fails? I wasn't planning on redundancy for it, should I? Similarly, let's say I run FreeNAS at first for virtue of simplicity, and months later want to migrate to OI. Can I just nuke the OS drive, install OI and import the ZFS pool?

Comments on the build are also welcome, especially concerning controller cards.

The summary at the top's outdated, though. At the moment, the specs would look like this:

Board: Intel S1200KP
CPU: Intel Core i3-2120T
16 Gigs of non-ECC DDR3-1333
Case: Lian Li PC-25Q
PSU: Silverstone ST50F-P or Corsair CX500
Storage: 1x 64GB Crucial m4 (OS); 4-6x 2TB Seagate Barracuda (storage); possibly 1x 20GB SLC SSD for ZIL (probably a later upgrade)
Controller: Intel SASUC8i, LSI 9211-8i or Supermicro AOC-USAS2-L8i (they're all based on LSI chips, and they're all mentioned favorably in various ZFS threads around the web)

gkontos · Apr 24, 2012

pineconez said:
Board: Intel S1200KP

CPU: Intel Core i3-2120T

16 Gigs of non-ECC DDR3-1333

Case: Lian Li PC-25Q

PSU: Silverstone ST50F-P or Corsair CX500

Storage: 1x 64GB Crucial m4 (OS); 4-6x 2TB Seagate Barracuda (storage); possibly 1x 20GB SLC SSD for ZIL (probably a later upgrade)

Controller: Intel SASUC8i, LSI 9211-8i or Supermicro AOC-USAS2-L8i (they're all based on LSI chips, and they're all mentioned favorably in various ZFS threads around the web)

Watch for that LSI controller, you will need to either go with FreeBSD 9.0-STABLE or manually compile the driver.

From a 9.0-RELEASE:

Code:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xdff3c000-0xdff3ffff,0xdff40000-0xdff7ffff irq 16 at device 0.0 on pci3
mps0: Firmware: 12.00.00.00

That was upgraded to 9.0-STABLE:

Code:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xdff3c000-0xdff3ffff,0xdff40000-0xdff7ffff irq 16 at device 0.0 on pci3
mps0: Firmware: 12.00.00.00, Driver: 13.00.00.00-fbsd

The rest:

You can easily use RAIDZ2 with the 6 Barracuda drives. Make sure you 4K align them.

OS, L2ARC and SWAP can all go to the SSD

phoenix · Apr 25, 2012

pineconez said:
First: How's the performance difference between RAID-Z2 and a pool of mirrored vdevs? I know the latter will have significantly better performance, but will it matter in a home environment (considering I'll have to at most saturate 1Gig Ethernet). On the other hand, I'd like to use [mirror] vdevs for simplicity: anytime I want to expand the server, I just need to put two drives in and add them.

Mirror vdevs give you the best IOps, the best expandability (just add another pair), the best ability to replace drives (attach new drive to mirror creating 3-way mirror, resilver, detach old drive; never lose redundancy), and various other benefits. However, you do sacrifice 50% of your drive space.

My home server originally had a 3-disk raidz1 using 160 GB SATA drives. It was pokey, but good enough. Expanded the pool by adding a mirror of 120 GB IDE drives. It was better, but still pokey. Added a USB-based L2ARC, and things sped up a bunch!

Later, replaced the pool using a pair of mirror vdevs, using 500 GB SATA drives, with the same USB-based L2ARC. Thought things would be much faster, but it actually felt slower than the old pool. On a hunch, I removed the L2ARC .. and things sped up IMMENSELY! The mirror vdevs without an L2ARC are faster than I can use.

Downloading torrents at just under 10 Mbps, streaming XviD and x264 videos to the XMBC box, compiling software in the background, the drives barely break a sweat, and I see 20-30 MBps of reads/writes to the individual drives in gstat(8).

And replacing a dying drive was so much simpler (and safer!), using the attach;resilver;detach method compared to the offline;remove;replace method needed for raidz vdevs.

Second: What happens with the storage drives if the OS drive fails?

If you use completely separate drives for the OS and the ZFS pool, then nothing happens to the pool.

You install a new OS drive, rebuild the OS, and then just "zpool import" the pool and carry on.

I wasn't planning on redundancy for it, should I?

Depends on what you store on the OS drive, and how often you back it up to the ZFS pool, and how much downtime you can handle to replace it.

Similarly, let's say I run FreeNAS at first for virtue of simplicity, and months later want to migrate to OI. Can I just nuke the OS drive, install OI and import the ZFS pool?

Yes. FreeBSD 8.3 and 9.0 support ZFSv28. FreeBSD 7.x-8.2 support ZFSv15. FreeNAS 0.7/8.x support (I believe) ZFSv15. OpenIndiana/Illumos/Nexenta/etc support ZFSv28. So long as the OS supports the pool version, it will import the pool.

IOW, if you build the pool on FreeNAS (ZFSv15), then you can import it anywhere. If you build the pool on FreeBSD 8.3+ (ZFSv28), then it can only be imported into OI/Illumos/Nexenta. And so on.

bbzz · Apr 25, 2012

phoenix said:
Mirror vdevs give you the best IOps, the best expandability (just add another pair), the best ability to replace drives (attach new drive to mirror creating 3-way mirror, resilver, detach old drive; never lose redundancy), and various other benefits. However, you do sacrifice 50% of your drive space.

My home server originally had a 3-disk raidz1 using 160 GB SATA drives. It was pokey, but good enough. Expanded the pool by adding a mirror of 120 GB IDE drives. It was better, but still pokey. Added a USB-based L2ARC, and things sped up a bunch!

Later, replaced the pool using a pair of mirror vdevs, using 500 GB SATA drives, with the same USB-based L2ARC. Thought things would be much faster, but it actually felt slower than the old pool. On a hunch, I removed the L2ARC .. and things sped up IMMENSELY! The mirror vdevs without an L2ARC are faster than I can use. Downloading torrents at just under 10 Mbps, streaming XviD and x264 videos to the XMBC box, compiling software in the background, the drives barely break a sweat, and I see 20-30 MBps of reads/writes to the individual drives in gstat(8).

And replacing a dying drive was so much simpler (and safer!), using the attach;resilver;detach method compared to the offline;remove;replace method needed for raidz vdevs.

For my 6x2TB drives I went with raidz-2. I'm not sure I made right choice. I got extra 2 TB but, the main reason I went at the time was the fact that raidz-2 allows you to blow up any of two drives in a pool. Whereas with mirrored pools, failure of both drives in same vdev and your whole pool is gone. Still, I think as long as a Gbps link is saturated that's all that matters really.

gkontos · Apr 25, 2012

bbzz said:
For my 6x2TB drives I went with raidz-2. I'm not sure I made right choice. I got extra 2 TB but, the main reason I went at the time was the fact that raidz-2 allows you to blow up any of two drives in a pool. Whereas with mirrored pools, failure of both drives in same vdev and your whole pool is gone. Still, I think as long as a Gbps link is saturated that's all that matters really.

Plus you get the advantage of not being afraid loosing another drive during resilvering.

pineconez · Apr 25, 2012

gkontos said:
Plus you get the advantage of not being afraid loosing another drive during resilvering.

That'd not really matter though, would it? Unless the drive happened to be in the same mirror vdev as the one that failed first, then you're obviously screwed.

If I'm going to put the ZIL on the OS drive, I'd a) be stressing the SSD seriously, and b) should definitely mirror it (or else possibly lose recent writes if it crashes), correct? (Sadly, even a small 20GB SLC SSD is likely outside the budget at first, two of them most certainly are).

I'd like to use RAID-Z2, because it's more efficient and I'm guessing could saturate GigE effortlessly, but the expandability kills it for me. Going to go with mirrored vdevs. 8)

Four 2TB drives and one hot spare was what I was thinking about to begin with. Though I hope your performance issues were because of the USB connection, not because mirrored vdevs have a general problem with L2ARC?

gkontos · Apr 25, 2012

pineconez said:
That'd not really matter though, would it? Unless the drive happened to be in the same mirror vdev as the one that failed first, then you're obviously screwed.

RAIDz2 and RAIDz1 are different from mirrors. All vdevs belong to the same pool.

bbzz · Apr 25, 2012

pineconez said:
That'd not really matter though, would it? Unless the drive happened to be in the same mirror vdev as the one that failed first, then you're obviously screwed.

Exactly; I guess the chances of something like that are slim, but if it does happen, the whole pool is gone. Efficiency-wise, raidz-2 on 6 drives is not all that big of a space saver, I'm still loosing about 33%. I guess 3-way mirrored vdev would be super cool and super expensive.

I never really tested my performance as I figured I don't have a choice, but, can someone look at this and decipher for me? What exactly is what here? I repeated test two times.

Configuration is Intel i7 920, 12GB RAM, 6x2TB Samsung Spinpoint F4, that's one zpool. Another is 1x500GB WD Black set with copies=2 for OS. Also note I use geli AEX-256 on this. SATA controllers are just onboard (probably should get separate controller here?). There's no L2ARC, ZIL, or SSDs.

# sudo bonnie++ -d /storage/test -u root -s24576m

Code:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
puffy02.home.domain 24G   131  98 96009  18 57618  15   333  90 159431  17 122.1   2
Latency               153ms     756ms    1888ms     336ms     411ms    1363ms
Version  1.96       ------Sequential Create------ --------Random Create--------
puffy02.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23605  94 +++++ +++ 24890  97 23615  96 +++++ +++ 15393  80
Latency             12883us      98us     124us   25662us      32us   20943us
1.96,1.96,puffy02.home.domain,1,1335437890,24G,,131,98,96009,18,57618,15,333,90,159431,17,122.1,2,16,,,,,23605,94,
+++++,+++,24890,97,23615,96,+++++,+++,15393,80,153ms,756ms,1888ms,336ms,411ms,1363ms,12883us,98us,124us,25662us,32us
,20943us

Code:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
puffy02.home.domain 24G   127  96 92299  17 58540  15   335  92 188969  17 126.5   4
Latency               330ms     955ms    2163ms     292ms     418ms     804ms
Version  1.96       ------Sequential Create------ --------Random Create--------
puffy02.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24917  91 +++++ +++ 18969  93 23307  96 +++++ +++  5593  96
Latency              6406us     149us     247us   20333us      81us     303us
1.96,1.96,puffy02.home.domain,1,1335458182,24G,,127,96,92299,17,58540,15,335,92,188969,17,126.5,4,16,,,,,24917,91,
+++++,+++,18969,93,23307,96,+++++,+++,5593,96,330ms,955ms,2163ms,292ms,418ms,804ms,6406us,149us,247us,20333us,81us,
303us

pineconez · Apr 26, 2012

copies=X looks like mirroring on a filesystem basis. I didn't know you could do that. Way cool.

Okay, that answers my questions. I might show up here again in a couple of weeks though, if I run with FreeBSD. Thanks!

bbzz · Apr 26, 2012

Ok, now that I've completely hijacked this thread and taken hostages (and wasn't really trying to), can someone look at output I posted about performance of raidz-2, porfavor?