ZFS How do I create a zroot mirror by adding an extra disk?

F

froggit9000

I originally installed FreeBSD onto a single ZFS disk and now I want to create a 2-way ZFS mirror from this disk and another new disk and I want to make sure I do it properly.

Here is the current ZFS zroot boot pool:
Code: 
# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Wed Oct  8 01:45:21 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  nda4p4    ONLINE       0     0     0

errors: No known data errors

My thinking is that to do this I need to:
(1) replicate partition layout, types and sizes from the original disk to the new disk
(2) replicate boot code from the original disk to the new disk
(3) attach the new disk to the original disk to form a ZFS mirror
(4) update BIOS boot info to add the new disk to the boot disk list, so that if either half of the boot mirror dies, it will still be able to boot from the other surviving half of the mirror

(a) The original disk is nda4. This is M2_2 on the motherboard, a FreeBSD boot disk (M.2 NVMe SSD)
(b) The new disk is nda1. This is M2_1 on the motherboard, previously occupied by a Linux boot disk (M.2 NVMe SSD).

First I inspected the existing partition table on the original disk (nda4):
Code: 
# gpart show nda4
=>        40  1953525088  nda4  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)

Partition #2 is the FreeBSD boot code.
Partition #4 is the ZFS partition.

(1) To replicate the partition layout, types and sizes from original to new disk, is this correct?
Code: 
# gpart backup nda4 | gpart restore -F nda1

(2) To replicate the FreeBSD boot code from the original disk to the new disk, is this correct?
Code: 
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1

(3) To attach the new disk to the original disk to form a ZFS mirror, is this correct?
Code: 
# zpool attach zroot nda4p4 nda1p4

At this point, my understanding is that ZFS will resilver nda1p4 with the contents of nda4p4 and I can follow its progress status using zpool status zroot.

Once resilvering is complete I can move onto step (4) to add nda1 to the boot drive list in the BIOS.

Does all this look correct before I proceed?

I used the following 2 threads to glean the relevant info from:
https://forums.freebsd.org/threads/copying-partitioning-to-new-disk.60937/
https://forums.freebsd.org/threads/how-to-attach-new-device-to-existed-booting-zroot.98797/
 
So far, this seems to have worked, here is the log below for anyone also attempting the same:

Code: 
# camcontrol devlist
<Samsung SSD 990 PRO with Heatsink 1TB 3B2QJXD7>  at scbus5 target 0 lun 1 (pass3,nda1)
<Samsung SSD 990 PRO with Heatsink 1TB 3B2QJXD7>  at scbus8 target 0 lun 1 (pass6,nda4)

# gpart show nda4
=>        40  1953525088  nda4  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)

# gpart backup nda4 | gpart restore -F nda1

# gpart show nda1
=>        34  1953525101  nda1  GPT  (932G)
          34           6        - free -  (3.0K)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1423        - free -  (712K)

There are some slight differences shown in the partition info between nda4 and nda1 - identical make/model (Samsung 990 PRO) but bought a year apart, but same firmware version. Sizes seem to match but free space slightly different for some reason. Looking again, it seems the new disk has slighly more space: 1953525101 versus 1953525088)

Code: 
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1
partcode written to nda1p2
bootcode written to nda1

# zpool attach zroot nda4p4 nda1p4

# zpool status -v zroot
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Oct 10 22:14:56 2025
	30.7G / 30.7G scanned, 16.1G / 30.7G issued at 1.34G/s
	16.4G resilvered, 52.42% done, 00:00:10 to go
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0  (resilvering)

errors: No known data errors

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.6G in 00:00:23 with 0 errors on Fri Oct 10 22:15:19 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

Done, now time to reboot the machine, enter the BIOS and add ndap1 to the BIOS boot disk list.

In the BIOS, set the boot order to the following:
1st item: 'USB disk' so I can boot into my Linux system if the Linux M.2 NVMe SSD enclosure is connected to a USB C port.
2nd item: 'nda1' to boot into FreeBSD (one half of the zroot boot mirror)
3rd item: 'nda4' to boot into FreeBSD (the other half of the zroot boot mirror)

If all this works then job done.
 
Probably unnecessary, but thought I'd do a quick scrub of the pool just to be sure all OK:

Code: 
# zpool scrub zroot

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: scrub in progress since Fri Oct 10 23:09:38 2025
	30.7G / 30.7G scanned, 23.5G / 30.7G issued at 1.81G/s
	0B repaired, 76.51% done, 00:00:03 to go
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Fri Oct 10 23:09:57 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

All looks good :)

Now time to update the boot device order in the BIOS...
 
For some reason when I enter the BIOS in 'Advanced Mode' to set the boot order, it only allows me to select M2_2 (nda4) and M2_1 (nda1) is not shown and therefore not possible to select.

When I hit F7 to enter 'EZ mode' then M2_1 is listed as Samsung 990 PRO but is not selectable.

Hmmm strange.

My BIOS is from 2024-07 so I might try to get a later BIOS to see if that makes any difference.

However, in FreeBSD it seems the mirror is working, and I can boot using M2_2 (nda4) so even if that half of the mirror dies, I should be able to move the SSD in M2_1 slot into the M2_2 slot and reboot back into FreeBSD.

Will look into this more to see if I can find what's up.
 
In the BIOS it shows M2_2 device as 'UEFI OS (M2_2: Samsung SSD 990' so I looked back at the gpart info above and noticed partition #1 is marked as 'efi' so it got me thinking that as I copied nda4 (M2_2) to nda1 (M2_1), is there something I need to alter on partition #1 of nda1 so that the BIOS can recognise it properly as a UEFI device? Is the BIOS seeing identical info in partition #1 and therefore only recognising M2_2?
 
use glabel status — i bet that's the same partition as /dev/nda4p1, and thus is the EFI partition on your original disk.

your new disk needs a DOS filesystem created on the EFI partition, and the freebsd loader copied there. having multiple EFI partitions gets to be slightly annoying to manage by hand, thus we recommended the loaders-update way:
Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nvd1p1...
mount -t msdosfs /dev/nvd1p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt
Examining nvd0p1...
mount -t msdosfs /dev/nvd0p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nvd0p1 File(\EFI\BOOT\BOOTX64.EFI)
Updatable EFI loader: 2
-------------------------------
 
atax1a said:
use glabel status — i bet that's the same partition as /dev/nda4p1, and thus is the EFI partition on your original disk.

your new disk needs a DOS filesystem created on the EFI partition, and the freebsd loader copied there. having multiple EFI partitions gets to be slightly annoying to manage by hand, thus we recommended the loaders-update way:
Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nvd1p1...
mount -t msdosfs /dev/nvd1p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt
Examining nvd0p1...
mount -t msdosfs /dev/nvd0p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nvd0p1 File(\EFI\BOOT\BOOTX64.EFI)
Updatable EFI loader: 2
-------------------------------
Click to expand...

Yes, I think you're right:
Code: 
# glabel status
                    Name  Status  Components
            gpt/efiboot0     N/A  nda4p1
            gpt/gptboot0     N/A  nda4p2

I don't appear to have loaders-update installed - a find returns nothing:
Code: 
# find / -name loaders-update
#

It appears it's a port, but I don't have it on my system, so I suppose I have to get hold of it from fresh ports if I want to use it.
I found it here:
https://www.freshports.org/sysutils/loaders-update/

Or is another solution to install the msdos filesystem to nda1p1 and then copy over all the files from nda4p1?
 
you can use the binary package: pkg install loaders-update.

it doesn't make the filesystem, you'll have to do that first with newfs_msdos /dev/nda1p1 you might also wanna set a GPT label with gpart, too.

so, in summary:
Code: 
pkg install loaders-update
gpart modify -i 1 -l efiboot1 nda1
newfs_msdos /dev/nda1p1
loaders-update show-me
# check what it says against reality
loaders-update shoot-me
 
You must log in or register to reply here.
Back
Top