ZFS Replacing both disks in a zfs mirror

natterangell

natterangell

Alright, I need some help to ensure I get this right. I need to replace both SSDs in a mirror, because they're approacing the end of their wear level:
Code: 
gpart show
=>       40  500118112  ada0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1004K)
     534528   41943040     2  freebsd-swap  (20G)
   42477568  457639936     3  freebsd-zfs  (218G)
  500117504        648        - free -  (324K)

=>       40  500118112  ada1  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1004K)
     534528   41943040     2  freebsd-swap  (20G)
   42477568  457639936     3  freebsd-zfs  (218G)
  500117504        648        - free -  (324K)

Code: 
zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:06:30 with 0 errors on Mon Aug 17 18:20:48 2026
config:


    NAME            STATE     READ WRITE CKSUM
    zroot           ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        ada0p3.eli  ONLINE       0     0     0
        ada1p3.eli  ONLINE       0     0     0


errors: No known data errors

I boot via UEFI.

Code: 
efibootmgr -v
Boot to FW : false
BootCurrent: 0000
Timeout    : 1 seconds
BootOrder  : 0000, 0006, 0007, 0002, 0003, 0004, 0005
+Boot0000* FreeBSD HD(1,GPT,179f128f-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\FREEBSD\LOADER.EFI)
                      gpt/efiboot0:/EFI/FREEBSD/LOADER.EFI /boot/efi//EFI/FREEBSD/LOADER.EFI
 Boot0006* UEFI OS HD(1,GPT,179f128f-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
                      gpt/efiboot0:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
 Boot0007* UEFI OS HD(1,GPT,17ad7ae0-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
                      ada1p1:/EFI/BOOT/BOOTX64.EFI (null)

The disks I'm putting in are twice the size (512GB vs 256GB).

Finally, I should mention my current zfs-mirror is GELI encrypted (only the zfs partitions), I don't know how that fits into all this? The new pool should be encrypted as well.

I suppose zpool-replacerequires the new drives to be connected? I currently don't have any free sata ports, but I have an LSI card that I pass through to a bhyve VM. I can stop all VM's and Jails and disable the ppt device for the card if I need to have them connected.

In broad terms, am I correct that I should:
1. Partition the first drive to be replaced with similar EFI and swap partition, then create a zfs partition in the remaining space, init this partition with geli init, geli configure -b and geli attach, then proceed with zpool-replace? Will that work when specifying a larger volume? Or should the volume be the exact same size as the one it's replacing? According to the man page, I get the impression the size can be larger.
2. Add an UEFI boot entry for the new drive
3. Repeat the process for the last remaining original drive?

Alternatively, could I disconnect one drive and resilver to one of the new ones? And then remove the remaining drive and resilver again? I would still need GELI to play along.

Any pros and cons?
 
You can remove one of them and replace it with a new one. It's less than ideal as that means that you have a period where you've only got one drive, but I've done that recently. If you have the option, it's generally better to add a 3rd disk to the mirror and let it resilver before removing one of the ones that you're expecting to go bad and then repeat it a second time to remove the other one.
 
Yeah. I suppose there's very little risk though, both disks are still error free according to S.M.A.R.T. values. The other thing I've experienced with my motherboard before is that boot entries are removed if I move disks around, so replacing one disk with the one that is meant to stay in that port post-migration is beneficial.
 
Without the GELI aspect, what I've done in the past (this also works to expand a pool):
physically connect a new device
partition it, use gpart to look at current device partitioning and mirror it on new device
uze zpool attach to add the new device/partition to the existing mirror, making a 3 way mirror
let new device complete resilvering and shows as active in the 3 way mirror
then zpool detach to remove one of the old devices
repeat with a new device

Basically:
take 2 device mirror, add one to create 3 device mirror, let resilver, remove old one, add another to create 3 way mirror, remove old device. Now you have a 2 device mirror with devices that can be expanded so now you can zpool expand to get the extra space.

NOTE:
I've done this on mirrors that are not GELI. I don't know how GELI would fit into the steps.
 
mer said:
Without the GELI aspect, what I've done in the past (this also works to expand a pool):
physically connect a new device
partition it, use gpart to look at current device partitioning and mirror it on new device
uze zpool attach to add the new device/partition to the existing mirror, making a 3 way mirror
let new device complete resilvering and shows as active in the 3 way mirror
then zpool detach to remove one of the old devices
repeat with a new device

Basically:
take 2 device mirror, add one to create 3 device mirror, let resilver, remove old one, add another to create 3 way mirror, remove old device. Now you have a 2 device mirror with devices that can be expanded so now you can zpool expand to get the extra space.

NOTE:
I've done this on mirrors that are not GELI. I don't know how GELI would fit into the steps.
Click to expand...
In my experience, GELI doesn't really care. I did that remotely with the HDD I have colocated with zfs.rent and it wasn't an issue. As long as you have a partition/disk that's large enough to hold all the necessary data, which won't be a problem here if the replacement disks are 2x the original, then it works.
 
I have no experience about geli encrypted partition, but you have to do some works in order to copy the other partitions, namely efi and swap (and perhaps freebsd-boot). You can use gpart backup & restore. I would provide you a complete set of instructions, but the geli thing is stopping me.
 
hedwards thanks.
My point in noting that was I think GELI depends on "where/when" it's applied. On a whole device before doing zpool create I think think is different from gpart, zpool create on partitions, then apply GELI. I also think there may be "competing requirements" when things store stuff "in the last sector".
If GELI is applied first, it can use the whole physical sectors, then anything created underneath "disksize" is what GELI tells it.

That's why I said "I don't know because I have not done this, I've not done this because I've had no desire/need to on my personal systems".

Rereading the OP, it sounds like he did the GELI stuff on the freebsd-zfs partitions first and then zpool attach the GELI devices (as noted in the zpool status output).

Based on that I think the steps I have for nonGELI should work, just do the geli stuff before doing the zpool attach to create a 3 way mirror.

I also agree with Emrion about the efi partitions needing manual intervention.

Anyway, OP thanks for an interesting problem and others that are more GELI familiar than me for answers.
But OP if you can make a 3-way mirror you can protect yourself a bit from mucking things up (zpool attach vs zpool replace)
 
You must log in or register to reply here.
Back
Top