zpool advice for 6 disks

wszczep · Oct 25, 2012

Dear all,

This is my first post here on this forum though I was reading it for a quite time.

I need your advice how to arrange zpools for a file server where 6 disks will be attached.

There will be 2x SSD's for mirrored system zfs root, swap, and two partitions for L2ARC cache; and 4x 500GB HDD's for data pool.

For sure there will be root pool with two mirrored SSD drives. But later on shall I create:

one raidz-1 pool from 3x HDD's + one spare HDD
one raidz-2 pool from 4x HDD's
one pool from 2 mirrored 2x HDD's

in all cases there will be similar storage space at ca. 0,9TB

In case of a failure:

I can survive one HDD or SSD failure, where spare HDD will replace automatically;
I can survive two HDD failures and one SSD failure
I can survive two HDD failures and one SSD failure, but HDD failures must be from different mirror sets.

It may seems to be a theory problem, but please give me some advice.

Thanks in advance,
--
Wojciech

gkontos · Oct 25, 2012

Solution (a) is the most reliable with (c) being the most efficient in regards to throughput.

What would be the main purpose of the storage?

usdmatt · Oct 25, 2012

I'll just remind people that spares are NOT currently automatic on FreeBSD. You have to manually offline the failed drive and replace with the spare. This may affect the decision as you lose the 'minimal degraded time' benefit of a hot-spare drive. All you really gain with a ZFS spare on FreeBSD is the ability to start the replace without physical access to the machine (or having to wait days for a replacement if you don't have a cold-spare).

wblock@ · Oct 25, 2012

How would a three-drive RAIDZ1 plus a hot spare be more reliable than RAIDZ2 with four drives? It seems like RAIDZ2 is effectively (if not exactly) the same as RAIDZ1 plus an automatic hot spare that is already synchronized.

gkontos · Oct 25, 2012

@usdmatt,

You are right but having a hot spare standby is sometimes a life saver.

@wblock,

A RAIDZ2 with 4 drives has a terrible performance impact.

wszczep · Oct 25, 2012

usdmatt said:
I'll just remind people that spares are NOT currently automatic on FreeBSD. You have to manually offline the failed drive and replace with the spare. This may affect the decision as you lose the 'minimal degraded time' benefit of a hot-spare drive. All you really gain with a ZFS spare on FreeBSD is the ability to start the replace without physical access to the machine (or having to wait days for a replacement if you don't have a cold-spare).

This is unclear for me. The man zpool says:

Hot Spares
ZFS allows devices to be associated with pools as "hot spares". These
devices are not actively used in the pool, but when an active device
fails, it is automatically replaced by a hot spare.

In fact I have been affected few days ago with SSD, beeing a cache dev, crash. The disk was disconnected by the system from controller and marked as offline by zfs.
I thought that in case of HDD fail the spare dev will be activated and pool will be resilvered automatically.
Then in case a/ (one raidz-1 pool from 3x HDD's + one spare HDD) the spare can be assigned both to data and root pool.

I must test somehow this behaviuor, or maybe someone did it already?

wszczep · Oct 25, 2012

gkontos said:
What would be the main purpose of the storage?

This pool will be for small business office (15-20 workstations) purpose, mainly office documents, graphic files, mailboxes.

If spares would work automatically I would choose option a/

phoenix · Oct 25, 2012

FreeBSD does not support automatic resilver using spares in ZFS. This feature is not enabled on FreeBSD. The man page has not been updated to reflect that (in < 9.1).

And, spare disks can only be associated with a single pool at the moment. There's work ongoing in the OSS ZFS community to fix this.

wblock@ · Oct 25, 2012

gkontos said:
@wblock,

A RAIDZ2 with 4 drives has a terrible performance impact.

Okay, but for reliability, the online disk is, well, online.

gkontos · Oct 25, 2012

wszczep said:
This pool will be for small business office (15-20 workstations) purpose, mainly office documents, graphic files, mailboxes.

If spares would work automatically I would choose option a/

Spares don't work automatically but they are always good to be around.

IMHO it is much better to use 2 striped mirrors than a a RAIDZ2 with 4 disks.

The reason I suggested the spare in the first place is due to the fact that I ALWAYS keep the same amount of spare drives in RAIDZ configurations.

My SOHO server (much like yours) is consisted from a 3 X 1TB RAIDZ1. A similar capacity disk is always available and used as spare. So far it has been used 3 times during the past 4 years.

My clients storage servers range from RAIDZ1 to RAIDZ2. (minimum 6 disks, average 8) In all cases they keep spares.

And of course, there is always a different backup device depending on the budget of the client.

wszczep · Oct 30, 2012

Thx everybody for the help.

@gkontos, wblock
I will go for an option (c) i.e. two stripped mirrors + two mirrored SSD's for sys.
Additionally I will buy some additionall low budget SATA controller (eg. JMicron JMB363 chipset based card) to connect some disk as spare.

@phoenix, usdmatt
I hope that automatic spare replacement will be in ZFS in future. Though You said that "spare disks can only be associated with a single pool at the moment", I was able to assign the same spare (in fact gpt partition) to two different pools. But I haven't tried replace procedure yet.

Concerning zfs pool of two mirrored SSD's with swap and cache partitions on them. I wonder if following idea is good and can be implemented in real-life SOHO fileserver:

Code:

gpart create -s gpt ada0
gpart add -t freebsd-boot -a4k -b40   -s512k -l boot0  ada0
gpart add -t freebsd-zfs       -b2048 -s10G  -l sys0   ada0
gpart add -t freebsd-swap             -s2G   -l swap0  ada0
gpart add -t freebsd-ufs              -s20G  -l cache0 ada0
newfs -U -o time -t /dev/gpt/cache0

all above applied also to ada1

then send bootcode to both boot0 and boot1
create zfs pool mirror with gnop trick on sys0 and sys1
than after creating pool with two stripped mirrors add cache vdevs on cache0 and cache1

I mean will it be ok to have on SSD's mirrored sys on zfs, standard swap and L2ARC cache on stripped ufs partitions.

I hope You can understand me

gkontos · Oct 30, 2012

wszczep said:
Concerning zfs pool of two mirrored SSD's with swap and cache partitions on them. I wonder if following idea is good and can be implemented in real-life SOHO fileserver:

Code:

gpart create -s gpt ada0 gpart add -t freebsd-boot -a4k -b40 -s512k -l boot0 ada0 gpart add -t freebsd-zfs -b2048 -s10G -l sys0 ada0 gpart add -t freebsd-swap -s2G -l swap0 ada0 gpart add -t freebsd-ufs -s20G -l cache0 ada0 newfs -U -o time -t /dev/gpt/cache0

all above applied also to ada1

then send bootcode to both boot0 and boot1
create zfs pool mirror with gnop trick on sys0 and sys1
than after creating pool with two stripped mirrors add cache vdevs on cache0 and cache1

I mean will it be ok to have on SSD's mirrored sys on zfs, standard swap and L2ARC cache on stripped ufs partitions.

I hope You can understand me

Better use freebsd-zfs for your cache partition and use the gnop trick there too.

What I usually do in storages is a UFS boot from a USB stick. Or a UFS boot from an embedded PCI-X device. Then the SSDs are devided into SWAP, L2ARC and if required ZIL.

The boot devices are always installed with labels and cloned making a replacement very easy.

SWAP in never mirrored. (better performance)
L2ARC is also never mirrored (better performance)
ZIL is always mirrored!

In your case, I would suggest not to waste more than 8GB for the OS. This of course depends on the software you plan on installing but give or take. Then, split your SWAP to the SSD's and split the remaining space for L2ARC

In your case that would give you: 8GB for the OS, 4GB combined for SWAP and approximately 44GB combined for L2ARC

wszczep · Oct 31, 2012

gkontos said:
Better use freebsd-zfs for your cache partition and use the gnop trick there too.
[...]
L2ARC is also never mirrored (better performance)

Do You mean:

Code:

gnop create -S 4096 /dev/gpt/cache0
zpool create cachepool0 mirror gpt/cache0.nop gpt/some_fake_vdev
zpool export cachepool0
gnop destroy /dev/gpt/cache0.nop
zpool import cachepool0
zpool detach cachepool0 some_fake_vdev

leaving just that cache0 zfs partition in one vdev pool?

Or in other words, how to make a gnop trick on a single zfs partition? other posts I have read points gnop trick on at least two vdevs.

gkontos · Oct 31, 2012

wszczep said:
Do You mean:

Code:

gnop create -S 4096 /dev/gpt/cache0 zpool create cachepool0 mirror gpt/cache0.nop gpt/some_fake_vdev zpool export cachepool0 gnop destroy /dev/gpt/cache0.nop zpool import cachepool0 zpool detach cachepool0 some_fake_vdev

leaving just that cache0 zfs partition in one vdev pool?

Or in other words, how to make a gnop trick on a single zfs partition? other posts I have read points gnop trick on at least two vdevs.

[CMD=""]gnop create -S 4096 /dev/gpt/cache0[/CMD]
[CMD=""]gnop create -S 4096 /dev/gpt/cache1[/CMD]
[CMD=""]zpool add <pool> cache cache0.nop cache1.nop[/CMD]
[CMD=""]zpool export <pool>[/CMD]
[CMD=""]gnop destroy /dev/gpt/cache0.nop /dev/gpt/cache1.nop[/CMD]
[CMD=""]zpool import <pool>[/CMD]

Sfynx · Nov 5, 2012

gkontos said:
IMHO it is much better to use 2 striped mirrors than a a RAIDZ2 with 4 disks.

In that case the entire pool fails if two drives in a mirror fail, so for redundancy a raidz2 is better. With added L2ARC the performance might not be that big of an issue.