ZFS How do I create a zroot mirror by adding an extra disk?

froggit9000

froggit9000

I originally installed FreeBSD onto a single ZFS disk and now I want to create a 2-way ZFS mirror from this disk and another new disk and I want to make sure I do it properly.

Here is the current ZFS zroot boot pool:
Code: 
# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Wed Oct  8 01:45:21 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  nda4p4    ONLINE       0     0     0

errors: No known data errors

My thinking is that to do this I need to:
(1) replicate partition layout, types and sizes from the original disk to the new disk
(2) replicate boot code from the original disk to the new disk
(3) attach the new disk to the original disk to form a ZFS mirror
(4) update BIOS boot info to add the new disk to the boot disk list, so that if either half of the boot mirror dies, it will still be able to boot from the other surviving half of the mirror

(a) The original disk is nda4. This is M2_2 on the motherboard, a FreeBSD boot disk (M.2 NVMe SSD)
(b) The new disk is nda1. This is M2_1 on the motherboard, previously occupied by a Linux boot disk (M.2 NVMe SSD).

First I inspected the existing partition table on the original disk (nda4):
Code: 
# gpart show nda4
=>        40  1953525088  nda4  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)

Partition #2 is the FreeBSD boot code.
Partition #4 is the ZFS partition.

(1) To replicate the partition layout, types and sizes from original to new disk, is this correct?
Code: 
# gpart backup nda4 | gpart restore -F nda1

(2) To replicate the FreeBSD boot code from the original disk to the new disk, is this correct?
Code: 
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1

(3) To attach the new disk to the original disk to form a ZFS mirror, is this correct?
Code: 
# zpool attach zroot nda4p4 nda1p4

At this point, my understanding is that ZFS will resilver nda1p4 with the contents of nda4p4 and I can follow its progress status using zpool status zroot.

Once resilvering is complete I can move onto step (4) to add nda1 to the boot drive list in the BIOS.

Does all this look correct before I proceed?

I used the following 2 threads to glean the relevant info from:
https://forums.freebsd.org/threads/copying-partitioning-to-new-disk.60937/
https://forums.freebsd.org/threads/how-to-attach-new-device-to-existed-booting-zroot.98797/
 
So far, this seems to have worked, here is the log below for anyone also attempting the same:

Code: 
# camcontrol devlist
<Samsung SSD 990 PRO with Heatsink 1TB 3B2QJXD7>  at scbus5 target 0 lun 1 (pass3,nda1)
<Samsung SSD 990 PRO with Heatsink 1TB 3B2QJXD7>  at scbus8 target 0 lun 1 (pass6,nda4)

# gpart show nda4
=>        40  1953525088  nda4  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)

# gpart backup nda4 | gpart restore -F nda1

# gpart show nda1
=>        34  1953525101  nda1  GPT  (932G)
          34           6        - free -  (3.0K)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1423        - free -  (712K)

There are some slight differences shown in the partition info between nda4 and nda1 - identical make/model (Samsung 990 PRO) but bought a year apart, but same firmware version. Sizes seem to match but free space slightly different for some reason. Looking again, it seems the new disk has slighly more space: 1953525101 versus 1953525088)

Code: 
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1
partcode written to nda1p2
bootcode written to nda1

# zpool attach zroot nda4p4 nda1p4

# zpool status -v zroot
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Oct 10 22:14:56 2025
	30.7G / 30.7G scanned, 16.1G / 30.7G issued at 1.34G/s
	16.4G resilvered, 52.42% done, 00:00:10 to go
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0  (resilvering)

errors: No known data errors

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: resilvered 31.6G in 00:00:23 with 0 errors on Fri Oct 10 22:15:19 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

Done, now time to reboot the machine, enter the BIOS and add ndap1 to the BIOS boot disk list.

In the BIOS, set the boot order to the following:
1st item: 'USB disk' so I can boot into my Linux system if the Linux M.2 NVMe SSD enclosure is connected to a USB C port.
2nd item: 'nda1' to boot into FreeBSD (one half of the zroot boot mirror)
3rd item: 'nda4' to boot into FreeBSD (the other half of the zroot boot mirror)

If all this works then job done.
 
Probably unnecessary, but thought I'd do a quick scrub of the pool just to be sure all OK:

Code: 
# zpool scrub zroot

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: scrub in progress since Fri Oct 10 23:09:38 2025
	30.7G / 30.7G scanned, 23.5G / 30.7G issued at 1.81G/s
	0B repaired, 76.51% done, 00:00:03 to go
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

# zpool status -v zroot
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:19 with 0 errors on Fri Oct 10 23:09:57 2025
config:

	NAME        STATE     READ WRITE CKSUM
	zroot       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    nda4p4  ONLINE       0     0     0
	    nda1p4  ONLINE       0     0     0

errors: No known data errors

All looks good :)

Now time to update the boot device order in the BIOS...
 
For some reason when I enter the BIOS in 'Advanced Mode' to set the boot order, it only allows me to select M2_2 (nda4) and M2_1 (nda1) is not shown and therefore not possible to select.

When I hit F7 to enter 'EZ mode' then M2_1 is listed as Samsung 990 PRO but is not selectable.

Hmmm strange.

My BIOS is from 2024-07 so I might try to get a later BIOS to see if that makes any difference.

However, in FreeBSD it seems the mirror is working, and I can boot using M2_2 (nda4) so even if that half of the mirror dies, I should be able to move the SSD in M2_1 slot into the M2_2 slot and reboot back into FreeBSD.

Will look into this more to see if I can find what's up.
 
In the BIOS it shows M2_2 device as 'UEFI OS (M2_2: Samsung SSD 990' so I looked back at the gpart info above and noticed partition #1 is marked as 'efi' so it got me thinking that as I copied nda4 (M2_2) to nda1 (M2_1), is there something I need to alter on partition #1 of nda1 so that the BIOS can recognise it properly as a UEFI device? Is the BIOS seeing identical info in partition #1 and therefore only recognising M2_2?
 
use glabel status — i bet that's the same partition as /dev/nda4p1, and thus is the EFI partition on your original disk.

your new disk needs a DOS filesystem created on the EFI partition, and the freebsd loader copied there. having multiple EFI partitions gets to be slightly annoying to manage by hand, thus we recommended the loaders-update way:
Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nvd1p1...
mount -t msdosfs /dev/nvd1p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt
Examining nvd0p1...
mount -t msdosfs /dev/nvd0p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nvd0p1 File(\EFI\BOOT\BOOTX64.EFI)
Updatable EFI loader: 2
-------------------------------
 
atax1a said:
use glabel status — i bet that's the same partition as /dev/nda4p1, and thus is the EFI partition on your original disk.

your new disk needs a DOS filesystem created on the EFI partition, and the freebsd loader copied there. having multiple EFI partitions gets to be slightly annoying to manage by hand, thus we recommended the loaders-update way:
Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nvd1p1...
mount -t msdosfs /dev/nvd1p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt
Examining nvd0p1...
mount -t msdosfs /dev/nvd0p1 /mnt
Would run: cp /boot/loader.efi /mnt/EFI/boot/bootx64.efi
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nvd0p1 File(\EFI\BOOT\BOOTX64.EFI)
Updatable EFI loader: 2
-------------------------------
Click to expand...

Yes, I think you're right:
Code: 
# glabel status
                    Name  Status  Components
            gpt/efiboot0     N/A  nda4p1
            gpt/gptboot0     N/A  nda4p2

I don't appear to have loaders-update installed - a find returns nothing:
Code: 
# find / -name loaders-update
#

It appears it's a port, but I don't have it on my system, so I suppose I have to get hold of it from fresh ports if I want to use it.
I found it here:
https://www.freshports.org/sysutils/loaders-update/

Or is another solution to install the msdos filesystem to nda1p1 and then copy over all the files from nda4p1?
 
you can use the binary package: pkg install loaders-update.

it doesn't make the filesystem, you'll have to do that first with newfs_msdos /dev/nda1p1 you might also wanna set a GPT label with gpart, too.

so, in summary:
Code: 
pkg install loaders-update
gpart modify -i 1 -l efiboot1 nda1
newfs_msdos /dev/nda1p1
loaders-update show-me
# check what it says against reality
loaders-update shoot-me
 
I executed all the commands you listed above, but loaders-update shoot-me doesn't seem to copy over the boot code from nda4p1 to nda1p1, so I don't know what I'm doing wrong.

I will show the output of loaders-update show-me and loaders-update shoot-me below:

Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nda4p1...
Efi partition nda4p1 is already mounted in /boot/efi.
EFI loader /boot/efi/efi/freebsd/loader.efi is up-to-date.
EFI loader /boot/efi/efi/boot/bootx64.efi is up-to-date.
Examining nda1p1...
mount -t msdosfs /dev/nda1p1 /mnt
There is no FreeBSD loader in nda1p1
umount /mnt

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda4...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda4p2 is up-to-date.
Examining nda1...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda1p2 is up-to-date.

-------------------------------
Your current boot method is UEFI.
Boot device: nda4p1 File(\EFI\BOOT\BOOTX64.EFI)
One or more target partition(s) have been found...
All loaders are up-to-date.
-------------------------------

Code: 
# loaders-update shoot-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nda4p1...
Efi partition nda4p1 is already mounted in /boot/efi.
EFI loader /boot/efi/efi/freebsd/loader.efi is up-to-date.
EFI loader /boot/efi/efi/boot/bootx64.efi is up-to-date.
Examining nda1p1...
mount -t msdosfs /dev/nda1p1 /mnt
There is no FreeBSD loader in nda1p1
umount /mnt

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda4...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda4p2 is up-to-date.
Examining nda1...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda1p2 is up-to-date.

-------------------------------
Your current boot method is UEFI.
Boot device: nda4p1 File(\EFI\BOOT\BOOTX64.EFI)
One or more target partition(s) have been found...
All loaders are up-to-date.
-------------------------------

I assumed that after creating the msdos filesystem on nda1p1 that running loaders-update shoot-me would copy all boot files over from nda4p1 to nda1p1 but maybe this is not the case.

Do I have to manually copy all the files over from nda4p1 to nda1p1?

# loaders-update --version
loaders-update v1.2.1
 
atax1a said:
hm, yeah, i guess you must manually copy them the first time! sorry for the misinformation.
Click to expand...

OK, thanks.

It does look like the https://www.freshports.org/sysutils/loaders-update/ can copy the boot loader code to another partition, as the man page for sysutils/loaders-update shoot-me states:

Code: 
This mode may update the loader(s), but ask for confirmation before each one.  You need to be root.

'may' seems to be the operative term, so it suggests that it can update the boot loader, so maybe I am missing something here - it seems like you shouldn't have to manually copy the file tree, so maybe the author, emrion can shine a little light on this? Of course, if it does not copy the boot loader file tree from one partition to another I will manually copy across the files from nda4p1 to nda1p1.
 
sysutils/loaders-update, as the name suggests, only updates the loaders. It doesn't copy a loader where there was no FreeBSD loader before. In this case, you need to create the suited directories and copy manually /boot/loader.efi in two places: nda1p1 /efi/freebsd/loader.efi and nda1p1 /efi/boot/bootx64.efi.

You see, a software like this cannot decide that this efi partition is used by FreeBSD. It may be used by another OS (i.e. multiboot system).
 
Emrion said:
sysutils/loaders-update, as the name suggests, only updates the loaders. It doesn't copy a loader where there was no FreeBSD loader before. In this case, you need to create the suited directories and copy manually /boot/loader.efi in two places: nda1p1 /efi/freebsd/loader.efi and nda1p1 /efi/boot/bootx64.efi.

You see, a software like this cannot decide that this efi partition is used by FreeBSD. It may be used by another OS (i.e. multiboot system).
Click to expand...

Ok, thanks emrion.

So if I have understood, something like this should work?:

Code: 
# gpart modify -i 1 -l efiboot1 nda1
# newfs_msdos /dev/nda1p1

# mount -t msdosfs /dev/nda1p1 /mnt

# mkdir -p /mnt/efi/freebsd
# mkdir -p /mnt/efi/boot
# cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
# cp /boot/loader.efi /mnt/efi/boot/bootx64.efi

# loaders-update show-me  (see what is displayed)
# loaders-update shoot-me (updates the loader on nda1p1 so that it can boot from nda1p1)
 
After running those commands, I think things have worked because now the nda1 NVMe SSD appears in the BIOS as a selectable boot disk.

I disabled the previous boot disk nda4 and enabled the nda1 disk as the boot disk and saved the BIOS changes, rebooted into FreeBSD then queried the system and below I show what appears.

Code: 
# loaders-update show-me
loaders-update v1.2.1

One or more efi partition(s) have been found.

Examining nda4p1...
Efi partition nda4p1 is already mounted in /boot/efi.
EFI loader /boot/efi/efi/freebsd/loader.efi is up-to-date.
EFI loader /boot/efi/efi/boot/bootx64.efi is up-to-date.
Examining nda1p1...
mount -t msdosfs /dev/nda1p1 /mnt
EFI loader /mnt/efi/freebsd/loader.efi is up-to-date.
EFI loader /mnt/efi/boot/bootx64.efi is up-to-date.
umount /mnt

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda4...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda4p2 is up-to-date.
Examining nda1...
The pmbr on this disk is up-to-date.
The freebsd-boot partition nda1p2 is up-to-date.

-------------------------------
Your current boot method is UEFI.
Boot device: nda1p1 File(\EFI\BOOT\BOOTX64.EFI)
One or more target partition(s) have been found...
All loaders are up-to-date.
-------------------------------

My understanding is that due to the new boot code installed on nda1p1 and setting the BIOS to boot from nda1, it did boot from nda1 so that's a good step forward.

But looking above at the output from loaders-update show-me, it shows the following:

Code: 
Efi partition nda4p1 is already mounted in /boot/efi.

This is because /etc/fstab still refers to nda4p1.

So I looked at the contents of /etc/fstab:

Code: 
# cat /etc/fstab
# Device			Mountpoint	FStype	Options		Dump	Pass#
/dev/gpt/efiboot0	/boot/efi	msdosfs	rw			2		2
/dev/nda4p3			none		swap	sw			0		0
proc				/proc		procfs	rw			0		0

I see that the /boot/efi mountpoint refers to device /dev/gpt/efiboot0 which is nda4p1 as far as I know:

Code: 
# ls -l /dev/gpt
total 0
crw-r-----  1 root operator 0x98 Oct 15 00:29 efiboot0
crw-r-----  1 root operator 0xc3 Oct 15 00:29 efiboot1
crw-r-----  1 root operator 0x99 Oct 15 00:29 gptboot0

The /dev/gpt/efiboot1 item refers to nda1p1 as that is the label I set for the nda1p1 partition above.

To be able to boot from nda1p1 or nda4p1, I now need to specify both the nda1 and nda4 SSDs within the BIOS, which should be easy to do.

But...as nda1 and nda4 are part of a 2-way mirror (at least the zroot ZFS partition is), to make FreeBSD boot reliably from either nda1p1 OR nda4p1, what do I need to specify within /etc/fstab for the /boot/efi mountpoint? (considering that either disk may be inaccesible due to disk failure)
 
we would, actually, avoid mounting either one. the only thing that lives there is the loader, and it only needs to be mounted when you're mucking with it — the loader is configured by files in /boot. remove them both from /etc/fstab and let loaders-update handle it.

(this is, of course, what we do)
 
Yes, your nda1p1 is functional. That said, you didn't need to use newfs as nda1p1 was already formatted (because if not, loaders-update would complain about that).

At some point, the FreeBSD installer used fstab to automatically mount the main efi partition (with the idea to one day automatically update its loader, I think). It wasn't the case before. There is no point to have such a partition mounted. It could even be dangerous because the efi content is more easily accessible. But, to be frank, that's not a big deal.

loaders-update is able to face both situations.

The boot process has nothing to do with the mounting of the efi partition. Indeed, the boot process is finished up before the partition is mounted.
 
atax1a said:
we would, actually, avoid mounting either one. the only thing that lives there is the loader, and it only needs to be mounted when you're mucking with it — the loader is configured by files in /boot. remove them both from /etc/fstab and let loaders-update handle it.

(this is, of course, what we do)
Click to expand...

Thanks - I will comment out that line then.
 
Emrion said:
Yes, your nda1p1 is functional. That said, you didn't need to use newfs as nda1p1 was already formatted (because if not, loaders-update would complain about that).

At some point, the FreeBSD installer used fstab to automatically mount the main efi partition (with the idea to one day automatically update its loader, I think). It wasn't the case before. There is no point to have such a partition mounted. It could even be dangerous because the efi content is more easily accessible. But, to be frank, that's not a big deal.

loaders-update is able to face both situations.

The boot process has nothing to do with the mounting of the efi partition. Indeed, the boot process is finished up before the partition is mounted.
Click to expand...

Yes, I only put it as it didn't harm as there was nothing on that partition and I wanted to collect all the commands together in case I need to do something similar again one day.

OK great, so it sounds like the mountpoint is a relic and maybe should be removed from the installation process, as it is no longer useful perhaps - or at least commented out with a comment to re-enable the line if required for admin purposes?

Thanks for your help with this.

Finally, is it worth adding an extra mountpoint for the swap partition? I already have /dev/nda4p3 none swap sw 0 0 but maybe I should add a line for /dev/nda1p3 too?
 
You must log in or register to reply here.
Back
Top