ZFS Opinion: 2-drive mirror most capable of recovery (?)

[Avery Freeman]

Hi,

Are 2-drive mirrors really the easiest pools to recover one can find in the wild? If recovery capability is paramount to all other considerations, should it indefinitely be the best option?

I'm asking this question because I recently had a 4-drive z-mirror fail to import, it was reported as being in a degraded stated (faulted) and I really want to avoid this happening again

I came along this post where Argentum shares his strategy of using only 2-drive mirrors for this reason, which I'm surprised I'd never heard of or considered previously: https://forums.freebsd.org/threads/performance-raidz1-vs-mirroring.78061/#post-487054

1. Considering that the rotating disk space is cheap these days, I can easily buy sufficiently large disk and mirror it;
2. ZFS mirror itself is faster than RAIDZ. It is also more convenient to use. Easy to replicate the system, by moving one disk out of mirror to new platform;
3. Small and fast SSD-s are also very cheap these days. Addting L2ARC and SLOG to the ZFS pool improves the speed significantly, giving a feeling that the whole pool is SSD;
4. Also, on desktop computers, SATA ports are limited resource. So, considering that, with my current motherboard, I have populated two ports with rotating HDD and two ports with small (128G) SSD-s with freebsd-zfs partitions for L2ARC and SLOG mirror over these two SSD-i.

So my takeaway from this is if you have only a mirror, and no striping is involved (like say, 4 drives in 2 striped mirrors) it is much easier to recover, the mention he made about replicating with a physical drive, etc.

What are people's opinions on this, and have you any experience you'd like to share?

Also, what if there's a log device involved, is recovery and/or replication using a physical drive just as easy, or does that throw a wrench in things?

 

[Alain De Vos]

It all depends on how much disk space you want to waste.
The more you waste , e.g. a zfs-mirror wastes 50%, the easier/faster recovery.
Waste less, eg RAIDZ(1) with 10 drives will waste maybe only 10%. (How to calculate exactly ?)
But on recovery all the 9 disks will have to be read and ok to reconstruct the 10th.

If you lose a log device your filesystem will be in a "previous state". Previous as all data that was not in the log device.
I don't see log devices on spinning disks very useful as there will be no speed gain.

 

[astyle]

I'm frankly sold on using SSD's for just about everything. I still have a 256 MB USB stick that I got before 2005, it's been re-formatted who knows how many times, and it's still usable. By comparison, a spinning platter will degrade pretty bad after about 5-7 years. Never tried a RAID-Z config (Simply found it easier for my personal scenarios to just buy a few more high-capacity storage drives), but based on my reading about it in the past (no links, sorry!), I concluded that striping should be pretty reliable.

 

[Avery Freeman]

I'm frankly sold on using SSD's for just about everything. I still have a 256 MB USB stick that I got before 2005, it's been re-formatted who knows how many times, and it's still usable. By comparison, a spinning platter will degrade pretty bad after about 5-7 years. Never tried a RAID-Z config (Simply found it easier for my personal scenarios to just buy a few more high-capacity storage drives), but based on my reading about it in the past (no links, sorry!), I concluded that striping should be pretty reliable.

Wow, I've totally had USB flash drives and SD cards die. I've been using USB drives for booting ESXi since 2015 and they're only written when changing settings or during upgrades, since the OS/hypervisor is entirely run from memory, and even one of those has died. Plus a couple of personal-use USB flash drives (Sandisk 4GB and 8GB 'Cruzer Fit' model) I was loading OS on for installations.

You must just be lucky ?

I've had really good luck with HGST drives that are noted as being low-MBTF on backblaze's reports, I beat the hell out of 4x 7K2000s from 2011 recording TV to them 24x7 for 3-4 years, in one system a fan stopped working and a couple drives reported max temps of over 80deg C, I replaced them with some 4TB HGSTs for about 2 years but recently started recording TV on them again. Still work just fine. But I even have a couple of the notorious WD Greens 2TBs from 2011 - I disabled park and enabled TLER and I am using them for recording security cameras ? they last a long time if they're not parking 100s of times a day.

The only problem I have personally had is zpool import issues on striped mirrors. I just wonder if one of the drives had failed and it was only a mirror instead of a striped-mirror, if I still would have been able to import it.

 

[Avery Freeman]

It all depends on how much disk space you want to waste.
The more you waste , e.g. a zfs-mirror wastes 50%, the easier/faster recovery.
Waste less, eg RAIDZ(1) with 10 drives will waste maybe only 10%. (How to calculate exactly ?)
But on recovery all the 9 disks will have to be read and ok to reconstruct the 10th.

If you lose a log device your filesystem will be in a "previous state". Previous as all data that was not in the log device.
I don't see log devices on spinning disks very useful as there will be no speed gain.

I'm not talking about "wasting space" at all. The question is what's the easiest to recover. There are plenty of discussions about which method of creating a pool gives the most usable storage area, but what about people's recovery experiences?

Having just been burned on a 4-drive striped mirror, I'm wondering if I would have been able to more easily recover from two discrete mirrored drive pools.

 

[astyle]

I'm not talking about "wasting space" at all. The question is what's the easiest to recover. There are plenty of discussions about which method of creating a pool gives the most usable storage area, but what about people's recovery experiences?

Having just been burned on a 4-drive striped mirror, I'm wondering if I would have been able to more easily recover from two discrete mirrored drive pools.

It could be the brand of your disks. Samsung, Western Digital, Seagate - I try to stick with known quality brands. I never had a problem with SanDisk stuff, but that was the brand I saw mentioned most often when people complained about USB sticks dying. Sometimes, it helps to pay a little extra for quality stuff. Last time I had any trouble recovering a file - that was before 2005 from a 3M floppy disk

 

[Alain De Vos]

I'm not talking about "wasting space" at all. The question is what's the easiest to recover. There are plenty of discussions about which method of creating a pool gives the most usable storage area, but what about people's recovery experiences?

Having just been burned on a 4-drive striped mirror, I'm wondering if I would have been able to more easily recover from two discrete mirrored drive pools.

Mirrors are fastest to recover i thought that was clear.
But if during the "resilvering" the good disk in the mirror fails you lose the whole pool.
A raidz3 has no problems if two disks fail.

Note : zfs can "stripe" over two mirrors but not mirror two "striped".

 

[mer]

Having just been burned on a 4-drive striped mirror,

Just to make sure I understand this:
4 drives total, 2 mirrors of 2 drives, then the 2 mirrors striped together?

If so then just like any "stripe" the disks (in this case the 2 mirror vdevs) are concatenated without redundancy.
Like any stripe if lose one of the disks the whole pool is shot. That would mean losing both disks in one of your mirror vdevs.

Recovery: In theory mirrors should be the easiest to recover; it depends on "what" you can do with the hardware.
If you have enough space in the system (physical connectivity like power, SATA, etc) you can attach a disk to the degraded mirror, giving you a 3way mirror (single mirror vdev consisting of 3 physical devices), let the new device resilver and then detach the failed device. It's a bit of a workaround for what Alain De Vos points out in #7 because as long as it's not a hard failure the pool can usually limp along until the resilver is done.

 

[Argentum]

I'm not talking about "wasting space" at all. The question is what's the easiest to recover. There are plenty of discussions about which method of creating a pool gives the most usable storage area, but what about people's recovery experiences?

Having just been burned on a 4-drive striped mirror, I'm wondering if I would have been able to more easily recover from two discrete mirrored drive pools.

It all depends on the application and my post describes desktop use-case. Mirror on desktop is excellent because of the ease you can clone your system. Also grow the storage capacity on the fly when needed.