Solved Two RAIDZ1 on 4 disks - will this increase performance?

P

Peacekeeper2000

A currently theoretical question after I read the book "FreeBSD Mastery: ZFS" as this was not discussed in the book:
Instead of using the full xTB disk as VDEV and build a RAIDz1 with 4 of them, I could partition the disks in x/2TB VDEV partitions and build two RAID1z with 4 VDEV's each. When I pool the 2 RAIDz1 under one root, I would have the same amount of space ( nearly) . The redundancy would be the same ( 1 HD failure wouldn't be a problem).
Code: 
storage
 raidz1-0
  gpt/disk0p1
  gpt/disk1p1
  gpt/disk2p1
  gpt/disk3p1
 raidz1-1
  gpt/disk0p2
  gpt/disk1p2
  gpt/disk2p2
  gpt/disk3p2

Would the performance increase compared to a single RAIDz1 with 4 disks? Would zpool scrub storage be faster as the size of each RAID1z is smaller ? Or would that only be , when you build two pools ( Storage 1 and Storage 2 with each a single Raidz1 ) ? With such a split I would loose a bit the flexibility of ZFS datasets, but with xTB disks the pool size is anyhow already very big.
Any opinion and comment is welcome - beside that this is not the biggest problem of the world ;)
 
In general, advice for zfs says to use full disks for building vdevs, not disk partitions.
The reason for that is that the most likely bottleneck (at least for spinning rust devices) is how fast you can write / read one physical device.
If you partition one physical device into two partitions, and use them in the same vdev, you have just doubled the amount of data that need to be written "simultaneously" onto the physical device.
Not an ideal situation.
 
No, you will actually get worse performance if you do this.

ZFS will try to stripe reads across the two vdevs, which means it will try to write data to the first half of the disk, and then the second half of the disk, at the same time! And, if you are trying to read from the pool, ZFS will have to read from the first half of the disk, and the second half of the disk, at the same time. You'll be thrashing the heads back and forth, causing latency to go up, and IOps to go down.

Adding more vdevs to a pool is almost always a good idea, if you want to increase performance. But don't try to use the same disks multiple times within the same pool. Just don't.
 
Generally, more vdevs, better performance, yes. However this post is about multiple vdevs using partitions on the same device and I highly doubt that will perform better. With standard disks it'll probably be much worse as ZFS will likely write to different locations of the disk in each vdev, so as it's striping across the two vdevs, it will be constantly seeking back and forth on the disks. With SSDs you wouldn't have any seek problems, but I doubt it would be faster than just having the whole disks in one vdev.

I can't see size of the vdevs affecting scrubs much either in this instance. Scrub speed is mainly affected by the total amount of data, and how fragmented it is. Striping the data across multiple partitions on the same disks is likely to effectively make the data more fragmented. You're reading the same amount of data off the same disks but more spread out.
 
Thanks to all for your replies !
I see the trouble with the spins as ZFS strips over both raids - and I remember from a customer project 15 years ago where we used more smaller disks just to increase the amount of spindles while we striped LVM over them.

usdmatt said:
...
I can't see size of the vdevs affecting scrubs much either in this instance. Scrub speed is mainly affected by the total amount of data, and how fragmented it is. Striping the data across multiple partitions on the same disks is likely to effectively make the data more fragmented. You're reading the same amount of data off the same disks but more spread out.
Click to expand...

Ok, that helped to sort my thinking. So even if I split the pool in to pools
Code: 
storage1_used
raidz1-0
  gpt/disk0p1
  gpt/disk1p1
  gpt/disk2p1
  gpt/disk3p1
storage2_yet_unused
raidz1-0
  gpt/disk0p2
  gpt/disk1p2
  gpt/disk2p2
  gpt/disk3p2
I would not get a better scrub turnaround as scrub anyhow only verifies written datasets and not empty parts of the pool - right ?
 
You can use multiple partitions on disks just fine if you use them for separate pools, like in your last example. But only if you do not use the two pools at the same time, otherwise you would force the disks to seek between the two positions, which drastically lowers performance.

ZFS indeed checks only stored data when scrubbing, and not empty data. That is why scrubbing an almost empty pool completes very quickly, though a very full pool consisting of many small files will take a bit longer. But it doesn't just check parity consistency, but also integrity of stored data - whether the checksum of the file matches the on-disk stored data.
 
You must log in or register to reply here.
Back
Top