ZFS Suggestions on upgrading mirrored zpool to striped mirror

[stratacast1]

I'm at a point where I will need to add storage to one of my zpools soon and I'm wanting to go from my standard 2-disk mirror to a 4-disk striped mirror. I know I could simply chuck 2 drives in and add them to the pool as a mirrored vdev, but I'm finding that since ZFS doesn't rebalance data, the only real gains I'll get is in added capacity. To my understanding, if I were to add 2 disks, strip them, send the data over to this new pool and turn my old pool into mirrored drives in the new one, I'd get some added performance. What do you all think? I've been reading around and these are the 2 prominent options. The least painful is definitely adding the 2 drives as a new mirrored vdev to my pool from what I see. I wouldn't mind getting some extra performance if it's actually a reasonable amount.

Currently, I have 2 HGST 4 KiB drives, and would likely need to get WD Red Pros at this point to get a matching 7200RPM speed and I see they also support 4KiB blocks.

 

[`Orum]

...since ZFS doesn't rebalance data, the only real gains I'll get is in added capacity.

Yes, this is true, initially. If you read/write/delete data on the mirror frequently, it will eventually rebalance. But if this is just a pool that you dump things on and rarely delete from, it's probably not the best situation for performance.

If you want to go from a 2 drive mirror to a 2x2 mirror configuration, and perfectly evenly distribute the data, I think the only way to do that is by making sure all drives are exactly the same size, backing up your data elsewhere (or not reusing your current drives), recreating the pool from scratch, and moving the data back on. Tedious and inconvenient, yes, but that would probably get you the best performance.

Out of curiosity, is there a reason you want a 2x2 ZFS config? The only reason I can see for this style of configuration is high read IOPS with fault tolerance, as ZFS isn't known for its write speed, and for raw sequential reads you'd probably do just as well with a RAID-Z2 config, but have better redundancy (any two drives failing instead of max 1 per vdev). You could also add a L2ARC device to further improve read IOPS, but its usefulness will depend heavily on your situation.

 

[ralphbsz]

With striping, you get two things: Better read throughput in MB/s for large files (since alternating stripes of the file can be read from the two different disks), and twice as much throughput (IO/s a.k.a. IOps) for random reads in large files and for small files. Obviously, this only works if the large file was written split over both disks (not before you configured striping), and if the working set of the random workload is split over both disks.

But the question is: Can you actually make use of that? Let's start with throughput. With a single disk, you already get 100 MB/s or more. Many processes that do some interesting work with data are not capable of consuming data that fast. So the only way to get data consumed that fast is to have multiple processes running in parallel. Do you expect to do this? The same situation applies to random IO (or reading small files): A single process will nearly always be only doing one IO at a time. In that case, having two disks that are striping doesn't help, since one will usually be idle.

What I'm really saying is: the benefit of striping only happens for parallel workloads, which most single-user systems for amateur workloads don't have. So maybe just don't worry about it?

 

[stratacast1]

Out of curiosity, is there a reason you want a 2x2 ZFS config?

Mostly what ralphbsz said. You'd get better performance with certain workloads. Though, what ralphbsz said and with some more consideration, I likely wouldn't even benefit much from it. Especially to rebuild my zpool to get a balanced pool.

The other reasons I want striped mirrors is in this case, I can just slap in 2 drives and create another mirror of them in my zpool. No rebuilding needed. The other aspect is a rebuild of a degraded pool takes much less time to resilver and much less load since there is no calculation of parity bits.

Perhaps for the sake of simplicity and me likely not being able to benefit from it like I thought. I'll probably just add another mirror to my zpool

 

[`Orum]

twice as much throughput (IO/s a.k.a. IOps) for random reads

While it's true you'd get twice as many IOPS with striping the vdevs over a RAID-Z2 config, IOPS should not be confused with throughput. Throughput depends on not only how many IOPS are occurring, but also how much data is transferred per IO.

The other aspect is a rebuild of a degraded pool takes much less time to resilver and much less load since there is no calculation of parity bits.

Rebuild speed is governed by write performance, so I don't think you'll see any difference there. The only exception might be in a RAID-Z2 configuration, where two drives fail, vs a single drive failing in each mirror. Even then, if they both resilver at the same time, I'm not sure you'd see any difference.

As for calculation of parity increasing load, this is negligible, unless you are using very fast drives (i.e. NVME) and slow CPUs (e.g. older Atoms), but it'd be quite weird to pair those together in the first place.

 

[stratacast1]

At this point I don't think I'd be able to take full advantage of RAID-Z2 on this host in terms of the increased throughput. The one thing I don't like about the RAID-Z2 idea is I'd either have to buy 4 new disks to make my pool and set my current 2 on the shelf for a rainy day, or I'd buy the 2 disks I'm already thinking about, blow away my pool, make my new zpool and restore from backup. With striped mirrors I could just add a new mirrored vdev, or create a striped vdev -> zfs send the data over -> stop all activity on my pool -> zfs send one last time -> destroy old zpool -> mirror the stripe with my old disks with minimal downtime