ZFS create a mirror of 4-disks stripes

fluca1978 · Sep 25, 2025

Given 8 identical geoms, how can I create a mirror of two 4-disks stripes?
The following creates four mirrors of two disks:

Code:

% doas zpool create tank mirror gpt/disco1 gpt/disco2  mirror gpt/disco3 gpt/disco4  mirror gpt/disco5 gpt /disco6  mirror gpt/disco7 gpt/disco8

% zpool status                                   
  pool: tank
 state: ONLINE
config:

        NAME            STATE     READ WRITE CKSUM
        tank            ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/disco1  ONLINE       0     0     0
            gpt/disco2  ONLINE       0     0     0
          mirror-1      ONLINE       0     0     0
            gpt/disco3  ONLINE       0     0     0
            gpt/disco4  ONLINE       0     0     0
          mirror-2      ONLINE       0     0     0
            gpt/disco5  ONLINE       0     0     0
            gpt/disco6  ONLINE       0     0     0
          mirror-3      ONLINE       0     0     0
            gpt/disco7  ONLINE       0     0     0
gpt/disco8 ONLINE 0 0 0

while I would like to have a situation like:

Code:

        NAME            STATE     READ WRITE CKSUM
        tank            ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/disco1  ONLINE       0     0     0
            gpt/disco2  ONLINE       0     0     0
            gpt/disco3  ONLINE       0     0     0
            gpt/disco4  ONLINE       0     0     0
          mirror-1      ONLINE       0     0     0
            gpt/disco5  ONLINE       0     0     0
            gpt/disco6  ONLINE       0     0     0
            gpt/disco7  ONLINE       0     0     0
            gpt/disco8  ONLINE       0     0     0

SirDice · Sep 25, 2025

Too risky. Suppose one of the disks in mirror-0 (1 to 4) dies, then you've lost the entire half of the mirror. Any disk failing on the 'other' side (5 through 8) would destroy the entire pool.

I get what you want to do, but the way you presented it made it a striped set of 2 4 way mirrors.

Maturin · Sep 25, 2025

Personally I wouldn't recommend stripes at this number of drives.
I would recommend you consider a raidz3 pool.
This way you can afford to lose 3 drives, and you get more storage to use, as if you mirror that much.

Check out a ZFS pool calculator for the details, for example this one:
ZFS Capacity Calculator on TrueNAS.com

covacat · Sep 25, 2025

my understanding is that the 4 way mirror version will withstand 6 drives loss in the best case and 4 3 (thx Maturin ) in the worse case

Code:

 NAME            STATE     READ WRITE CKSUM
        tank            ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/disco1  ONLINE       0     0     0
            gpt/disco2  ONLINE       0     0     0
            gpt/disco3  ONLINE       0     0     0
            gpt/disco4  ONLINE       0     0     0
          mirror-1      ONLINE       0     0     0
            gpt/disco5  ONLINE       0     0     0
            gpt/disco6  ONLINE       0     0     0
            gpt/disco7  ONLINE       0     0     0
            gpt/disco8  ONLINE       0     0     0

fluca1978 · Sep 25, 2025

Thanks all for suggestions, while I understand this is not the recommended way of using storage, I was curious to understand what is the syntax to achieve such configuration.

covacat · Sep 25, 2025

Code:

for  i in $(jot 8);do dd if=/dev/zero of=faked_$i bs=1m count=66;done
zpool create lala mirror /home/titus/faked_[1-4] mirror /home/titus/faked_[5-8]

zpool status lala
  pool: lala
 state: ONLINE
config:

    NAME                     STATE     READ WRITE CKSUM
    lala                     ONLINE       0     0     0
      mirror-0               ONLINE       0     0     0
        /home/titus/faked_1  ONLINE       0     0     0
        /home/titus/faked_2  ONLINE       0     0     0
        /home/titus/faked_3  ONLINE       0     0     0
        /home/titus/faked_4  ONLINE       0     0     0
      mirror-1               ONLINE       0     0     0
        /home/titus/faked_5  ONLINE       0     0     0
        /home/titus/faked_6  ONLINE       0     0     0
        /home/titus/faked_7  ONLINE       0     0     0
        /home/titus/faked_8  ONLINE       0     0     0


zpool detach  lala /home/titus/faked_7
zpool detach  lala /home/titus/faked_1
zpool detach  lala /home/titus/faked_3

zpool status lala
  pool: lala
 state: ONLINE
config:

    NAME                     STATE     READ WRITE CKSUM
    lala                     ONLINE       0     0     0
      mirror-0               ONLINE       0     0     0
        /home/titus/faked_2  ONLINE       0     0     0
        /home/titus/faked_4  ONLINE       0     0     0
      mirror-1               ONLINE       0     0     0
        /home/titus/faked_5  ONLINE       0     0     0
        /home/titus/faked_6  ONLINE       0     0     0
        /home/titus/faked_8  ONLINE       0     0     0

Maturin · Sep 25, 2025

covacat said:
4 in the worse case

3
Because mirror-0 and mirror-1 are ment to be in a stripe. When all 4 drives of one mirror are toast at the same time, that mirror is killed which killes the stripe.

A stripe is killed when one chain link fails. If those consist of mirrors the maximum amount of drives you can afford to lose is the number of drives of each mirror - 1
to deal with "worst - best case" suggests a security you don't really have, since you always have to reckon statistic distributions, which means Murphy's law

.

fluca1978 · Friday at 8:14 AM

covacat said:

Code:

for  i in $(jot 8);do dd if=/dev/zero of=faked_$i bs=1m count=66;done
zpool create lala mirror /home/titus/faked_[1-4] mirror /home/titus/faked_[5-8]

zpool status lala
  pool: lala
 state: ONLINE
config:

    NAME                     STATE     READ WRITE CKSUM
    lala                     ONLINE       0     0     0
      mirror-0               ONLINE       0     0     0
        /home/titus/faked_1  ONLINE       0     0     0
        /home/titus/faked_2  ONLINE       0     0     0
        /home/titus/faked_3  ONLINE       0     0     0
        /home/titus/faked_4  ONLINE       0     0     0
      mirror-1               ONLINE       0     0     0
        /home/titus/faked_5  ONLINE       0     0     0
        /home/titus/faked_6  ONLINE       0     0     0
        /home/titus/faked_7  ONLINE       0     0     0
        /home/titus/faked_8  ONLINE       0     0     0


zpool detach  lala /home/titus/faked_7
zpool detach  lala /home/titus/faked_1
zpool detach  lala /home/titus/faked_3

zpool status lala
  pool: lala
 state: ONLINE
config:

    NAME                     STATE     READ WRITE CKSUM
    lala                     ONLINE       0     0     0
      mirror-0               ONLINE       0     0     0
        /home/titus/faked_2  ONLINE       0     0     0
        /home/titus/faked_4  ONLINE       0     0     0
      mirror-1               ONLINE       0     0     0
        /home/titus/faked_5  ONLINE       0     0     0
        /home/titus/faked_6  ONLINE       0     0     0
        /home/titus/faked_8  ONLINE       0     0     0

While this provides the situation as I wanted it to be, it turned out the situation I described was wrong.
The above creates two mirrors, with 4 disks mirroring each other, while I want a mirror made by two 4 disks stripes. Another layer is needed, something llike:

Code:

    lala                     ONLINE       0     0     0
      mirror-0               ONLINE       0     0     0
       raidz1-0
        /home/titus/faked_1  ONLINE       0     0     0
        /home/titus/faked_2  ONLINE       0     0     0
        /home/titus/faked_3  ONLINE       0     0     0
        /home/titus/faked_4  ONLINE       0     0     0
       raidz1-1
        /home/titus/faked_5  ONLINE       0     0     0
        /home/titus/faked_6  ONLINE       0     0     0
        /home/titus/faked_7  ONLINE       0     0     0
        /home/titus/faked_8  ONLINE       0     0     0

Maturin · Friday at 9:09 AM

fluca1978 said:
I want a mirror made by two 4 disks stripes.

That's no good idea!
This way you can afford only one disk to fail. This ones breaks the stripe, so the according mirror.
If the second disk fails in the other mirror, before you replaced the first one - don't underestimate the load you burden the disks for resilvering

- both stripes break, so you whole pool is toast.

This way you get the storage of 4 disks with the redundancy of 1 disk to fail.
With pooling all 8 disks into one raidz2, or a raidz3 pool you were way better off:
more storage, and more redundancy.
I strongly recommend to check on those options before you decide.

covacat · Friday at 9:11 AM

well by stripe he means raidz1 so 2 disks may fail / one for each raidz1

Maturin · Friday at 9:22 AM

yeah, ...well,... to me a stripe is a stripe, and a raidz1 is a raidz1.
While AFAIK "our" ZFS supports:
stripe, mirror, raid10, raidz1, raidz2, and raidz3
But okay, I don't have to bother at all. My 15TB raidz2 NAS works for several years so far.
I'm out. Good luck!

covacat · Friday at 9:22 AM

you can create 4 gmirror devices with gmirror or graid and zraid1 those 4 gmirrors
then you can lose 3 drives and still be ok. you can lose 5 in a favorable scenario

sko · Friday at 9:35 AM

fluca1978 said:
The above creates two mirrors, with 4 disks mirroring each other, while I want a mirror made by two 4 disks stripes. Another layer is needed, something llike:

That's not how zfs works.

Andriy · Friday at 11:45 AM

fluca1978 said:
while I would like to have a situation like:

What you showed there was two 4-way mirrors.
Do you really want that?

And, FYI, ZFS does not support any complex topologies.
A pool is always an array of top-level VDEVs where is VDEV can be a single disk, a mirror of disks, a RAIDZ.
Mirrors and RAIDZ-s can be composed only of simple disks (partitions, more generally).

Of course, you can play some GEOM topology tricks underneath ZFS.

Andriy · Friday at 11:50 AM

SirDice said:
Too risky. Suppose one of the disks in mirror-0 (1 to 4) dies, then you've lost the entire half of the mirror. Any disk failing on the 'other' side (5 through 8) would destroy the entire pool.

What the OP shown as a hypothetical zpool status output was actually two 4-way mirrors, so it wasn't what they said they wanted.
And what they wanted is not possible with ZFS.
ZFS can do only mirrors of disks (partitions), it cannot do mirrors of "stripes".

SirDice · Friday at 12:47 PM

Andriy said:
What the OP shown as a hypothetical zpool status output was actually two 4-way mirrors, so it wasn't what they said they wanted.

Yes, I know.

SirDice said:
I get what you want to do, but the way you presented it made it a striped set of 2 4 way mirrors.

VladiBG · Friday at 1:34 PM

Raid0+1 is not supported.

ZFS create a mirror of 4-disks stripes

fluca1978

SirDice

Administrator

Maturin

covacat

fluca1978

covacat

Maturin

fluca1978

Maturin

covacat

Maturin

covacat

sko

Andriy

Andriy

SirDice

Administrator

VladiBG