ZFS Replacing both disks in a zfs mirror

natterangell

natterangell

Alright, I need some help to ensure I get this right. I need to replace both SSDs in a mirror, because they're approacing the end of their wear level:
Code: 
gpart show
=>       40  500118112  ada0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1004K)
     534528   41943040     2  freebsd-swap  (20G)
   42477568  457639936     3  freebsd-zfs  (218G)
  500117504        648        - free -  (324K)

=>       40  500118112  ada1  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       2008        - free -  (1004K)
     534528   41943040     2  freebsd-swap  (20G)
   42477568  457639936     3  freebsd-zfs  (218G)
  500117504        648        - free -  (324K)

Code: 
zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:06:30 with 0 errors on Mon Aug 17 18:20:48 2026
config:


    NAME            STATE     READ WRITE CKSUM
    zroot           ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        ada0p3.eli  ONLINE       0     0     0
        ada1p3.eli  ONLINE       0     0     0


errors: No known data errors

I boot via UEFI.

Code: 
efibootmgr -v
Boot to FW : false
BootCurrent: 0000
Timeout    : 1 seconds
BootOrder  : 0000, 0006, 0007, 0002, 0003, 0004, 0005
+Boot0000* FreeBSD HD(1,GPT,179f128f-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\FREEBSD\LOADER.EFI)
                      gpt/efiboot0:/EFI/FREEBSD/LOADER.EFI /boot/efi//EFI/FREEBSD/LOADER.EFI
 Boot0006* UEFI OS HD(1,GPT,179f128f-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
                      gpt/efiboot0:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
 Boot0007* UEFI OS HD(1,GPT,17ad7ae0-79f3-11f1-88d5-001b21f2d048,0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
                      ada1p1:/EFI/BOOT/BOOTX64.EFI (null)

The disks I'm putting in are twice the size (512GB vs 256GB).

Finally, I should mention my current zfs-mirror is GELI encrypted (only the zfs partitions), I don't know how that fits into all this? The new pool should be encrypted as well.

I suppose zpool-replacerequires the new drives to be connected? I currently don't have any free sata ports, but I have an LSI card that I pass through to a bhyve VM. I can stop all VM's and Jails and disable the ppt device for the card if I need to have them connected.

In broad terms, am I correct that I should:
1. Partition the first drive to be replaced with similar EFI and swap partition, then create a zfs partition in the remaining space, init this partition with geli init, geli configure -b and geli attach, then proceed with zpool-replace? Will that work when specifying a larger volume? Or should the volume be the exact same size as the one it's replacing? According to the man page, I get the impression the size can be larger.
2. Add an UEFI boot entry for the new drive
3. Repeat the process for the last remaining original drive?

Alternatively, could I disconnect one drive and resilver to one of the new ones? And then remove the remaining drive and resilver again? I would still need GELI to play along.

Any pros and cons?
 
You can remove one of them and replace it with a new one. It's less than ideal as that means that you have a period where you've only got one drive, but I've done that recently. If you have the option, it's generally better to add a 3rd disk to the mirror and let it resilver before removing one of the ones that you're expecting to go bad and then repeat it a second time to remove the other one.
 
Yeah. I suppose there's very little risk though, both disks are still error free according to S.M.A.R.T. values. The other thing I've experienced with my motherboard before is that boot entries are removed if I move disks around, so replacing one disk with the one that is meant to stay in that port post-migration is beneficial.
 
Without the GELI aspect, what I've done in the past (this also works to expand a pool):
physically connect a new device
partition it, use gpart to look at current device partitioning and mirror it on new device
uze zpool attach to add the new device/partition to the existing mirror, making a 3 way mirror
let new device complete resilvering and shows as active in the 3 way mirror
then zpool detach to remove one of the old devices
repeat with a new device

Basically:
take 2 device mirror, add one to create 3 device mirror, let resilver, remove old one, add another to create 3 way mirror, remove old device. Now you have a 2 device mirror with devices that can be expanded so now you can zpool expand to get the extra space.

NOTE:
I've done this on mirrors that are not GELI. I don't know how GELI would fit into the steps.
 
mer said:
Without the GELI aspect, what I've done in the past (this also works to expand a pool):
physically connect a new device
partition it, use gpart to look at current device partitioning and mirror it on new device
uze zpool attach to add the new device/partition to the existing mirror, making a 3 way mirror
let new device complete resilvering and shows as active in the 3 way mirror
then zpool detach to remove one of the old devices
repeat with a new device

Basically:
take 2 device mirror, add one to create 3 device mirror, let resilver, remove old one, add another to create 3 way mirror, remove old device. Now you have a 2 device mirror with devices that can be expanded so now you can zpool expand to get the extra space.

NOTE:
I've done this on mirrors that are not GELI. I don't know how GELI would fit into the steps.
Click to expand...
In my experience, GELI doesn't really care. I did that remotely with the HDD I have colocated with zfs.rent and it wasn't an issue. As long as you have a partition/disk that's large enough to hold all the necessary data, which won't be a problem here if the replacement disks are 2x the original, then it works.
 
I have no experience about geli encrypted partition, but you have to do some works in order to copy the other partitions, namely efi and swap (and perhaps freebsd-boot). You can use gpart backup & restore. I would provide you a complete set of instructions, but the geli thing is stopping me.
 
hedwards thanks.
My point in noting that was I think GELI depends on "where/when" it's applied. On a whole device before doing zpool create I think think is different from gpart, zpool create on partitions, then apply GELI. I also think there may be "competing requirements" when things store stuff "in the last sector".
If GELI is applied first, it can use the whole physical sectors, then anything created underneath "disksize" is what GELI tells it.

That's why I said "I don't know because I have not done this, I've not done this because I've had no desire/need to on my personal systems".

Rereading the OP, it sounds like he did the GELI stuff on the freebsd-zfs partitions first and then zpool attach the GELI devices (as noted in the zpool status output).

Based on that I think the steps I have for nonGELI should work, just do the geli stuff before doing the zpool attach to create a 3 way mirror.

I also agree with Emrion about the efi partitions needing manual intervention.

Anyway, OP thanks for an interesting problem and others that are more GELI familiar than me for answers.
But OP if you can make a 3-way mirror you can protect yourself a bit from mucking things up (zpool attach vs zpool replace)
 
mer said:
hedwards thanks.
My point in noting that was I think GELI depends on "where/when" it's applied. On a whole device before doing zpool create I think think is different from gpart, zpool create on partitions, then apply GELI. I also think there may be "competing requirements" when things store stuff "in the last sector".
If GELI is applied first, it can use the whole physical sectors, then anything created underneath "disksize" is what GELI tells it.

That's why I said "I don't know because I have not done this, I've not done this because I've had no desire/need to on my personal systems".

Rereading the OP, it sounds like he did the GELI stuff on the freebsd-zfs partitions first and then zpool attach the GELI devices (as noted in the zpool status output).

Based on that I think the steps I have for nonGELI should work, just do the geli stuff before doing the zpool attach to create a 3 way mirror.

I also agree with Emrion about the efi partitions needing manual intervention.

Anyway, OP thanks for an interesting problem and others that are more GELI familiar than me for answers.
But OP if you can make a 3-way mirror you can protect yourself a bit from mucking things up (zpool attach vs zpool replace)
Click to expand...
No worries, I do the geli first and then add the resulting device to the zfs mirror and that's been working out fairly well for me.
 
natterangell said:
1. Partition the first drive to be replaced with similar EFI and swap partition, then create a zfs partition in the remaining space, init this partition with geli init, geli configure -b and geli attach, then proceed with zpool-replace?
Click to expand...
Emphasis here on geli configure -b. This is critical. Using the -b option would render the setup unbootable after replacing both disks. You need to use the -g option. See man page excerpt down below for details [1]. Unless, of course, you are using a unencrypted boot partition on a USB pendrive, for example. Then the -b option would make sense.

Also: you don't need to split the initialization and configuration of the geli(8) provider in two commands. This can be done in one.

E.g. (using menu-guided installation default options and values, except the -b option. The default is -bg, but -b is useless here):

You can view the provider flags of the existing providers calling geli list | grep -e Geom -e Flags . For all other provider information: geli list.

Use the same passphrase as on the original disks.

geli init -g -l 256 -s 4096 ada2p3 ada3p3


As for the rest of the disk replacement:
  • Recreate the partition scheme and partitions on the new disks. I would have said gpart backup ada0 | gpart restore ada2, but this method doesn't make a exact copy. See Bug 296208 - gpart(8): Backup/Create/Restore Differences . There is a workaround in the PR to avoid creating every single partition manually. Then gpart resize <freebsd-zfs> partition.
  • After creating the partitions, dd(1) the original disk working efi partition to both of the new disks.
  • Check /etc/fstab, make sure swap points to the correct partitions. Best is to use GPT labels (see gpart(8) how to label partitions if they are not labeled), e.g.:/dev/gpt/swap0 , /dev/gpt/swap1. If you want them encrypted: /dev/gpt/swap0.eli etc. "swap" devices in fstab with the .eli suffix are automatically initialized (encrypted), no manual configuration needed.
  • Eventually recreate the UEFI menu entry after replacing the first disk:

natterangell said:
Will that work when specifying a larger volume? Or should the volume be the exact same size as the one it's replacing? According to the man page, I get the impression the size can be larger.
Click to expand...
Correct: zpool-replace(8)
Rich (BB code): 
DESCRIPTION
     Replaces device with new-device. ...
 
     The size of new-device must be greater than or equal to the minimum size
     of all the devices in a mirror or raidz configuration.


[1] geli(8)
Code: 
     init       Initialize providers which need to be encrypted.
 
                -b                Try to decrypt this partition during boot,
                                  before the root partition is mounted.  This
                                  makes it possible to use an encrypted root
                                  partition.  One will still need bootable
                                  unencrypted storage with a /boot/ directory,
                                  which can be a CD-ROM disc or USB pen-drive,
                                  that can be removed after boot.


                -g                Enable booting from this encrypted root
                                  filesystem.  The boot loader prompts for the
                                  passphrase and loads loader(8) from the
                                  encrypted partition.
 
T-Daemon said:
Emphasis here on geli configure -b. This is critical. Using the -b option would render the setup unbootable after replacing both disks. You need to use the -g option. See man page excerpt down below for details [1]. Unless, of course, you are using a unencrypted boot partition on a USB pendrive, for example. Then the -b option would make sense.

Also: you don't need to split the initialization and configuration of the geli(8) provider in two commands. This can be done in one.

E.g. (using menu-guided installation default options and values, except the -b option. The default is -bg, but -b is useless here):

You can view the provider flags of the existing providers calling geli list | grep -e Geom -e Flags . For all other provider information: geli list.

Use the same passphrase as on the original disks.

geli init -g -l 256 -s 4096 ada2p3 ada3p3


As for the rest of the disk replacement:
  • Recreate the partition scheme and partitions on the new disks. I would have said gpart backup ada0 | gpart restore ada2, but this method doesn't make a exact copy. See Bug 296208 - gpart(8): Backup/Create/Restore Differences . There is a workaround in the PR to avoid creating every single partition manually. Then gpart resize <freebsd-zfs> partition.
  • After creating the partitions, dd(1) the original disk working efi partition to both of the new disks.
  • Check /etc/fstab, make sure swap points to the correct partitions. Best is to use GPT labels (see gpart(8) how to label partitions if they are not labeled), e.g.:/dev/gpt/swap0 , /dev/gpt/swap1. If you want them encrypted: /dev/gpt/swap0.eli etc. "swap" devices in fstab with the .eli suffix are automatically initialized (encrypted), no manual configuration needed.
  • Eventually recreate the UEFI menu entry after replacing the first disk:


Correct: zpool-replace(8)
Rich (BB code): 
DESCRIPTION
     Replaces device with new-device. ...
 
     The size of new-device must be greater than or equal to the minimum size
     of all the devices in a mirror or raidz configuration.


[1] geli(8)
Code: 
     init       Initialize providers which need to be encrypted.
 
                -b                Try to decrypt this partition during boot,
                                  before the root partition is mounted.  This
                                  makes it possible to use an encrypted root
                                  partition.  One will still need bootable
                                  unencrypted storage with a /boot/ directory,
                                  which can be a CD-ROM disc or USB pen-drive,
                                  that can be removed after boot.


                -g                Enable booting from this encrypted root
                                  filesystem.  The boot loader prompts for the
                                  passphrase and loads loader(8) from the
                                  encrypted partition.
Click to expand...
Thanks!

OK, so geli configure -b -g is the gist of it? Here are the current disks:
Code: 
geli list | grep -e Geom -e Flags
Geom name: ada0p3.eli
Flags: BOOT, GELIBOOT, AUTORESIZE
Geom name: ada1p3.eli
Flags: BOOT, GELIBOOT, AUTORESIZE

Am I correct that "-b" corresponds to "BOOT" and "-g" to "GELIBOOT"?

I get the impression the process is somewhat analoguous to linux when using LUKS, in that you first create the encrypted container, and then create a filesystem on the device mapper. The setup I have is basically what FreeBSD created for me installing the system.

EFI partitions are obviously unencrypted, so the manpage warning should be OK. I see no reason to copy or dd the EFI partitions, they only contain static files after all? Just create new partitions and copy the files over? Swap is less of a concern.
 
natterangell said:
Am I correct that "-b" corresponds to "BOOT" and "-g" to "GELIBOOT"?

I get the impression the process is somewhat analoguous to linux when using LUKS, in that you first create the encrypted container, and then create a filesystem on the device mapper. The setup I have is basically what FreeBSD created for me installing the system.

EFI partitions are obviously unencrypted, so the manpage warning should be OK. I see no reason to copy or dd the EFI partitions, they only contain static files after all? Just create new partitions and copy the files over? Swap is less of a concern.
Click to expand...
-b tells the system to try to decrypt the partition during boot. -g prompts for the passphrase and loads loader from the encrypted partition. I haven't used -g for quite a while as I just encrypt whatever it is that I want to encrypt on a separate volume so that when I need to do any major tinkering or upgrades, I can just completely remove all of those things and not have to worry about data loss.
 
mer said:
*[...]
But OP if you can make a 3-way mirror you can protect yourself a bit from mucking things up (zpool attach vs zpool replace)
Click to expand...
Thank you! I have a follow up question.

Consider the following two approaches:
1. Additional disk is added to create a 3-way mirror, as described so far. 1 of the original disks removed. Additional EFI boot entry created and then a reboot to check it works. If something went wrong with chosen GELI option or similar on new disk, I suppose the remaining original disk should boot. What is the status of the *removed* (no longer part of the mirror) disk in this scenario? It is not bootable, right? I suppose I should I reboot as a 3-way mirror to verify GELI correctly unlocks all three, and the pool is healthy before removing one of the original disks?

2. One of the original disks is removed, computer boots in "degraded" mirror state. New disk inserted, formatted and zfs resilvers. Something went wrong. In this scenario, the *removed* original disk should be able to boot the system on its own, right?

The reason I ask these stupid questions is that it's not fully clear to me how zfs figures out it is part of a pool or not.
 
natterangell said:
Thanks
Click to expand...
My pleasure.

natterangell said:
OK, so geli configure -b -g is the gist of it?
Click to expand...
Sure, but why not just: geli init -g ...? Skip the configure step ( unnecessary) and the -b option.

As I mentioned in my previous post, you can skip the -b option, since it doesn't serve a purpose in your current setup. I don't know why the /usr/libexec/bsdinstall/zfsboot script (this is the script executed during a menu-guided un-encrypted / encrypted Root-on-ZFS installation) has a geli(8) option [1], which has no logic (creating a unencrypted dedicated boot partition and populate it with a complete /boot directory) anywhere in the bsdinstall(8) scripts or binaries .

natterangell said:
Am I correct that "-b" corresponds to "BOOT" and "-g" to "GELIBOOT"?
Click to expand...
Yes, that's correct.

natterangell said:
I get the impression the process is somewhat analoguous to linux when using LUKS, in that you first create the encrypted container, and then create a filesystem on the device mapper.
Click to expand...
I don't much about LUKS and how to configure in detail, but from what I read in the past from Linux distro handbooks it's basically the same on Linux as it is on FreeBSD.

natterangell said:
I see no reason to copy or dd the EFI partitions,
Click to expand...
dd(1)'ing the efi partition is for convenience:
Code: 
# dd if=/dev/ada0p1 of=/dev/ada2p1 bs=1m
# dd if=/dev/ada0p1 of=/dev/ada3p1 bs=1m

Otherwise:
Code: 
# newfs_msdos -c1 -F32 /dev/ada2p1
# newfs_msdos -c1 -F32 /dev/ada3p1

# mount_msdosfs /dev/ada2p1 /mnt
# mkdir -p /mnt/efi/freebsd

# cp /boot/loader.efi /mnt/freebsd
# umount /mnt

# mount_msdosfs /dev/ada3p3 /mnt
# mkdir ...
# cp ..


[1]
Rich (BB code): 
202 GELI_PASSWORD_GELIBOOT_INIT='geli init -bg -e %s -J - -l 256 -s 4096 "%s"'
 
natterangell said:
The reason I ask these stupid questions is that it's not fully clear to me how zfs figures out it is part of a pool or not.
Click to expand...
That's a bit of "why" I was asking GELI first then zpool. A lot of these layers will write metadata/stuff in "the last sector of a partition".
So you do gpart, create a partition that runs from X to Y. That is raw device. If you do GELI on that now, GELI may use the last sector, so doing anything on the GELI-ized partition, the partition now runs from X to (Y minus 1 sector). You do zpool on that, the zpool vdev runs from X to (Y minus) and zpool uses the last sector of that for identifying stuff, so you have X to ((Y minus 1 sector) minus 1 sector) for data.
 
You must log in or register to reply here.
Back
Top