Solved bootloader issue on FreeBSD 15.0 -> 15.1 upgrade

dvl@

Developer
Hello,

I'm looking for someone with more bootloader knowledge and experience than I have. A plan appears at the end of this post.

I've been updating several FreeBSD 15.0 hosts to 15.1 using pkgbase. It concerns an EFI booting host.

The boot devices are not clearly shown here, as expected (based on https://www.freebsd.org/releases/15.1R/upgrading/#upgrade-loader-uefi)

Code:
[19:11 r730-01 root ~] # sysctl machdep.bootmethod
machdep.bootmethod: UEFI
[19:11 r730-01 root ~] # efibootmgr -v
BootCurrent: 0000
Timeout    : 0 seconds
BootOrder  : 0000, 0001
+Boot0000* EFI Fixed Disk Boot Device 1 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x4,0x0,0x0)/HD(1,GPT,6e0689af-ca20-11f0-9f70-ecf4bbefc954,0x800,0x64000)
 Boot0001  EFI Fixed Disk Boot Device 2 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x5,0x0,0x0)/HD(1,GPT,709f19af-ca20-11f0-9f70-ecf4bbefc954,0x800,0x64000)


Unreferenced Variables:
 Boot0004* Hard drive C: VenHw(d6c0639f-c705-4eb9-aa4f-5802d8823de6)
 Boot0005* IBA GE Slot 0100 v1588 BBS(0x80,IBA GE Slot 0100 v1588,0x0)

I also ran some magic mentioned by bsdimp (when I asked about another similar host issue):

Code:
[19:12 r730-01 root ~] # efivar cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath --utf8
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
\EFI\BOOT\BOOTX64.EFI

[19:14 r730-01 root ~] # sudo efivar cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderDev --utf8
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderDev
PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x4,0x0,0x0)/HD(1,GPT,6E0689AF-CA20-11F0-9F70-ECF4BBEFC954,0x800,0x64000)

That tells me it's booting from \EFI\BOOT\BOOTX64.EFI

I know ada0 and ada1 are my boot drives:

Code:
[19:19 r730-01 root ~] # zpool status zroot     
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:33 with 0 errors on Thu Jul 16 04:02:31 2026
config:

    NAME                               STATE     READ WRITE CKSUM
    zroot                              ONLINE       0     0     0
      mirror-0                         ONLINE       0     0     0
        gpt/zfs0_20170718AA0000185556  ONLINE       0     0     0
        gpt/zfs1_20170719AA1178164201  ONLINE       0     0     0

errors: No known data errors


=>       40  242255584  ada0  GPT  (116G)
         40       2008        - free -  (1004K)
       2048     409600     1  efi0_20170718AA0000185556  (200M)
     411648   16777216     2  swap0  (8.0G)
   17188864  225066760     3  zfs0_20170718AA0000185556  (107G)

=>       40  242255584  ada1  GPT  (116G)
         40       2008        - free -  (1004K)
       2048     409600     1  efi1_20170719AA1178164201  (200M)
     411648   16777216     2  swap1  (8.0G)
   17188864  225066760     3  zfs1_20170719AA1178164201  (107G)

With that in mind, I did this mount and looked around:

Code:
[19:18 r730-01 root ~] # mount_msdosfs /dev/ada0p1 /boot/efi
[19:19 r730-01 root ~] # ls -l /boot/efi
total 16
drwxr-xr-x  1 root wheel 8192 Dec  8  2025 EFI
[19:19 r730-01 root ~] # find /boot/efi
/boot/efi
/boot/efi/EFI
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/BOOTX64.EFI

That is consistent with what the efivar command output above.

With that in mind, my plan is to do the following:

  1. cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
  2. umount /boot/efi
  3. mount_msdosfs /dev/ada1p1 /boot/efi
  4. cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
  5. umount /boot/efi
  6. shutdown -r now
  7. profit
Does this seem reasonable?

I agree, the second umount[CMD] is not necessary.
 
Are ada0 and ada1 a mirror pair? I'm assuming they are, with a second partition the ZFS mirror.
Not that ada0 and ada1 are a geom mirror and ada[01]p1 are not mirrored.

If those assumptions are correct then I agree with your "plan" steps to update the efi partititions on the individual boot drives. I mildly disagree with "step #5 the second umount being unnecessary" (If I manually mount it I want to manually umount it before a reboot just because OCD)
 
With that in mind, my plan is to do the following:

  1. cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
  2. umount /boot/efi
That is consistent with the handbook. I don't automatically mount EFI in fstab but I do comment it out.
But yesterday I end up just manually mounting my Mirror EFI partition and running command #1

I dunno about ZFS mirror but I just did upgraded my firewall with gmirror pair of ssd and my RAID is "On the Mirror".
So I mount -t msdosfs /dev/mirror/gm0p1 /mnt and cp over loader.efi.
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi

Handbook had so many paths it messed me up. I finally just searched the mount and found it.

cmp /boot/efi/EFI/FREEBSD/LOADER.EFI /boot/efi/EFI/BOOT/BOOTX64.EFI
I misread this and was searching all over for/EFI/FREEBSD/
 
I dunno about ZFS mirror but I just did upgraded my firewall with gmirror pair of ssd and my RAID is "On the Mirror".
So I mount -t mosdosfs /dev/gm0p1 /mnt and cp over loader.efi.
Is gm0p1 a partition off the mirror? Not one of the devices in the mirror? If so then I think that's good.

I guess I'm asking (fake device names, and may not be correct):
system has ada0 and ada1. Do geom mirror ada0 ada1 so the "whole" device for ada0 and ada1 are mirrored. Lets say that mirror is named mirror1 so we get a /dev/mirror1 and I then do gpart /dev/mirror1 to create the efi partitions. Then if I mount /dev/mirror1p1 and copy to it the data goes to both devices because of GEOM.

I think Phishfry that is roughly what you are saying?

I guess "don't geom mirror devices, partition, then create zfs mirrors on the geom mirrored partitions"
 
My Mirror gets written to the EFI when I do the installation.
I drop out of installer and setup gmirror and format and return to installation.
So when the installer writes the EFI bits it goes into the gmirror UUID not a single disk.

Code:
 # efibootmgr -v
Boot to FW : false
BootCurrent: 0007
Timeout    : 1 seconds
BootOrder  : 0007, 000A, 0004
+Boot0007* UEFI OS HD(1,GPT,51cc9663-1b06-11ec-89b9-00e0ed5d51ce,0x28,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)
                      mirror/gm0p1:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
 Boot000A* UEFI: FiD 2.5 SATA10000 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x1,0xffff,0x0)/HD(1,GPT,51cc9663-1b06-11ec-89b9-00e0ed5d51ce,0x28,0x64000)
                                  VenHw(2d6447ef-3bc9-41a0-ac19-4d51d01b4ce6,300032003000310036003000300033004100410030004200300030003000300030003000340031000000)
 Boot0004  UEFI: Built-in EFI Shell  VenMedia(5023b95c-db26-429b-a648-bd47664c8012)


Unreferenced Variables:
 
My Mirror gets written to the EFI when I do the installation.
I drop out of installer and setup gmirror and format and return to installation.
So when the installer writes the EFI bits it goes into the gmirror UUID not a single disk.

Code:
 # efibootmgr -v
Boot to FW : false
BootCurrent: 0007
Timeout    : 1 seconds
BootOrder  : 0007, 000A, 0004
+Boot0007* UEFI OS HD(1,GPT,51cc9663-1b06-11ec-89b9-00e0ed5d51ce,0x28,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)
                      mirror/gm0p1:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
 Boot000A* UEFI: FiD 2.5 SATA10000 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x1,0xffff,0x0)/HD(1,GPT,51cc9663-1b06-11ec-89b9-00e0ed5d51ce,0x28,0x64000)
                                  VenHw(2d6447ef-3bc9-41a0-ac19-4d51d01b4ce6,300032003000310036003000300033004100410030004200300030003000300030003000340031000000)
 Boot0004  UEFI: Built-in EFI Shell  VenMedia(5023b95c-db26-429b-a648-bd47664c8012)


Unreferenced Variables:
Thanks for clarification. Sound almost like something from a Michael Lucas book :) (I have theoretical knowledge of doing that not practical)
 
i must admit I do refer to my only How-To when I create new arrays.

Code:
gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                      ada1 (ACTIVE)
Puny little Innodisk 16GB SSD. Very old. Smart does not work correctly.

But if I were Dan I would look at the BIOS and see what it is booting off of exactly.

Mine is set to "UEFI OS" which you can also see from efibootmgr.
Boot0007* UEFI OS HD
 
+Boot0000* EFI Fixed Disk Boot Device 1 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x4,0x0,0x0)/HD(1,GPT,6e0689af-ca20-11f0-9f70-ecf4bbefc954,0x800,0x64000)
Boot0001 EFI Fixed Disk Boot Device 2 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x5,0x0,0x0)/HD(1,GPT,709f19af-ca20-11f0-9f70-ecf4bbefc954,0x800,0x64000)<br>
Looking at Dans output it does look like the BIOS only see's individual drives.
Like I said I am clueless about ZFS but I would have assumed a BIOS EFI setting for /zroot/mirror-0/
 
Are ada0 and ada1 a mirror pair? I'm assuming they are, with a second partition the ZFS mirror.
Not that ada0 and ada1 are a geom mirror and ada[01]p1 are not mirrored.

Yes, they are each part of the zroot mirror.

If those assumptions are correct then I agree with your "plan" steps to update the efi partititions on the individual boot drives. I mildly disagree with "step #5 the second umount being unnecessary" (If I manually mount it I want to manually umount it before a reboot just because OCD)

Technically, I think the OS will umount them just fine, but yes, if you mount, umount.
 
  • Like
Reactions: mer
With that in mind, my plan is to do the following:
  1. cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
  2. umount /boot/efi
  3. mount_msdosfs /dev/ada1p1 /boot/efi
  4. cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
  5. umount /boot/efi
  6. shutdown -r now
  7. profit
Does this seem reasonable?

I agree, the second umount[CMD] is not necessary.

Here goes:


Code:
[21:42 r730-01 root ~] # cmp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI differ: char 158, line 3
[21:42 r730-01 root ~] # cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI

[21:42 r730-01 root ~] # umount /boot/efi

[21:42 r730-01 root ~] # mount_msdosfs /dev/ada1p1 /boot/efi
[21:42 r730-01 root ~] # cmp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI differ: char 158, line 3
[21:43 r730-01 root ~] # cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
[21:43 r730-01 root ~] # cmp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTX64.EFI
[21:43 r730-01 root ~] # umount /boot/efi

[21:43 r730-01 root ~] #

That proves both updates were required. They are no mirrors. They are separate entities.

reboot in progress.
 
reboot in progress.
Success.

Code:
[17:46 pro02 dvl ~] % r730
Last login: Fri Jul 17 21:42:13 2026 from pro02.startpoint.vpn.unixathome.org
[21:50 r730-01 dvl ~] % freebsd-version -ukr
15.1-RELEASE-p1
15.1-RELEASE-p1
15.1-RELEASE-p1
[21:50 r730-01 dvl ~] %

Thank you for coming to my TED talk.
 
OK cool.
Can we talk philosophical?

Shouldn't you really have a EFIBOOTMGR setting for zpool/mirror-0 ?
zpool get guid zroot That would get you the GUID to set into EFI.

You have two EFI partitions and I don't think you need to. Sure it works but it seems to irk me.
You are booting off a ZFS Pool or are you? Everybody should be in the pool. One EFI mirrored. Failure handled by ZFS pool.
Please anybody correct me if I am thinking wrongly.
 
What I am saying is this:
When SATA1 fails will BIOS rollover automatically?
Lets say your at a Phillies Game and you are updating your server over SSH and you do a reboot.
Ruttro.
Disk has failed at BIOS and is stalled....
Got to leave the game to attend to your server....
But if you were booting off "UEFI OS" via /zpool/mirror-0/ then BIOS only needs a live pool. Either Drive.
So what I am getting at is you can create new EFIBOOTMGR settings and with GUID you can point to the ZFS Pool.
Letting ZFS do its job of data redundancy.
That is my theory. Is anybody doing that? You also have BE and I don't know how they act. A pool is a pool.
 
I chuckle when I go searching for something and find I've already encountered it before....
This is one of the most ironic "I chuckle..." in a long time ;) (I ran across your ZFS on partitions a long time ago and followed it because it makes sense)

So back up in #9 and #10 that implies to me "standard partitioning". Separate devices, gpart on each that creates a EFI partitition, a swap (maybe) and a zfs partition. Then the zfs partitions (vdevs) are used to create a zfs mirror. That means the 2 efi partitions are separate so updates need to happen on each.
That is different from what I think Phishfry is doing (create GEOM mirror of raw devices, then partition the mirror)

I know one of my systems 2 devices, partitioned identically, but not geom mirror. boot partition, swap partition, zfs partition. zfs partitions made up a zfs mirror and I wound up forgetting to update BOTH boot partitions once. Bios decided to try and boot off the one I did not update, it did good until it went "oh crap the zfs mirror was zpool upgrade and I can't handle that". Totally my fault but a good lesson.
 
I call bullcrap on myself.
But if you were booting off "UEFI OS" via /zpool/mirror-0/ then BIOS only needs a live pool. Either Drive.
If BIOS stalls for bad disk the Mirror or SATA problem don't matter. POST check goes first. If POST stalls then you will be attending to the machine.
 
I periodically copy /boot/loader.efi to /boot/efi/efi/{boot/bootx64.efi,freebsd/loader.efi} to avoid this sort of problem, especially on some machines where I follow 15.1-STABLE or -CURRENT.

On my 15.1 machine:
$ md5 -r /boot/efi/efi/*/* /boot/loader.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/efi/efi/boot/bootx64.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/efi/efi/freebsd/loader.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/loader.efi
 
I think that when efibootmgr -v doesn't answer a path for the used loader, it's the default one that has been selected. It's /efi/boot/bootx64.efi for an amd64 arch.

Why didn't you try sysutils/loaders-update in show-me mode?
There is for longtime now, I have no question, but to type this command in shoot-me mode.

My server has two system disk in mirror like yours. I boot with efi. I have also the possibility to start in BIOS mode with two freebsd-boot partitions. So, the manual update was long. My utility does that in one second or so, just the time to confirm each loader update by pressing 'y'.
 
Why didn't you try sysutils/loaders-update in show-me mode?
Let's see:

Code:
[17:14 r730-01 dvl ~] % loaders-update 
loaders-update v1.4.0
Usage: loaders-update mode [-befgqry] [-m efi_mount_dir] [-s loaders_source_dir]
mode can be one of:
  show-me: just show the commands to type, change nothing.
  shoot-me: may update the loader(s), but ask for confirmation before each one.
Options:
  -b: scan only BIOS loaders (excludes EFI loaders).
  -e: scan only EFI loaders (excludes BIOS loaders).
  -f: won't check the freebsd-boot content for BIOS loaders update.
  -g: force to use 'gpart show' for disk detection.
  -q: quiet mode. No output to the console.
  -r: won't check the root file system for BIOS loaders update.
  -y: answer yes for shoot-me mode. Use with caution!
Return codes:
  0: no error.
  1: an error occured.
  2: at least one loader isn't up-to-date.
  3: = 1 + 2.

Hmm. As non-root:

Code:
[17:16 r730-01 dvl ~] % loaders-update show-me
loaders-update v1.4.0

One or more efi partition(s) have been found.

Examining ada1p1...
mount -t msdosfs /dev/ada1p1 /mnt
mount_msdosfs: /dev/ada1p1: Operation not permitted
Cannot mount ada1p1, so cannot looking for its loader(s).
Examining ada0p1...
mount -t msdosfs /dev/ada0p1 /mnt
mount_msdosfs: /dev/ada0p1: Operation not permitted
Cannot mount ada0p1, so cannot looking for its loader(s).

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda4...
grep: /dev/nda4p1: Permission denied
Error during access to /dev/nda4p1. Won't update these loaders.

-------------------------------
Your current boot method is UEFI.
One or more target partition(s) have been found...
But no loader seems to be updatable.
3 error(s) occured during the scan.
Consider to run this utility as root.
-------------------------------

OK, I'll pay this game:


Code:
[17:16 r730-01 dvl ~] % sudo loaders-update show-me
loaders-update v1.4.0

One or more efi partition(s) have been found.

Examining ada1p1...
mount -t msdosfs /dev/ada1p1 /mnt
EFI loader /mnt/EFI/BOOT/BOOTX64.EFI is up-to-date.
umount /mnt
Examining ada0p1...
mount -t msdosfs /dev/ada0p1 /mnt
EFI loader /mnt/EFI/BOOT/BOOTX64.EFI is up-to-date.
umount /mnt

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda4...
The pmbr on this disk is up-to-date.
Would run: gpart bootcode -p /boot/gptzfsboot -i 1 nda4

-------------------------------
Your current boot method is UEFI.
Boot device: Fixed Disk Boot Device 1 PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0x4,0x0,0x0)/HD(1,GPT,6e0689af-ca20-11f0-9f70-ecf4bbefc954,0x800,0x64000)
Updatable BIOS loader: 1
-------------------------------
[17:16 r730-01 dvl ~] %


nda4 is a red herring. It is not part of the boot process. Or should not be.

Code:
[17:16 r730-01 dvl ~] % gpart show nda4
=>       40  488397088  nda4  GPT  (233G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352  484200448     3  freebsd-zfs  (231G)
  488396800        328        - free -  (164K)
 
I periodically copy /boot/loader.efi to /boot/efi/efi/{boot/bootx64.efi,freebsd/loader.efi} to avoid this sort of problem, especially on some machines where I follow 15.1-STABLE or -CURRENT.

On my 15.1 machine:
$ md5 -r /boot/efi/efi/*/* /boot/loader.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/efi/efi/boot/bootx64.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/efi/efi/freebsd/loader.efi
3df7c6bbc56827cf6fab1da2091b31b1 /boot/loader.efi
I can't compete there. This is a very old host. With odd stuff.

Code:
[17:19 r730-01 dvl ~] % md5 -r /boot/efi/efi/*/* /boot/loader.efi

zsh: no matches found: /boot/efi/efi/*/*
 
OK cool.
Can we talk philosophical?

Shouldn't you really have a EFIBOOTMGR setting for zpool/mirror-0 ?
zpool get guid zroot That would get you the GUID to set into EFI.

Code:
[17:20 r730-01 dvl ~] % zpool get guid zroot
NAME   PROPERTY  VALUE                 SOURCE
zroot  guid      14192263921754489476  -

You have two EFI partitions and I don't think you need to. Sure it works but it seems to irk me.
You are booting off a ZFS Pool or are you? Everybody should be in the pool. One EFI mirrored. Failure handled by ZFS pool.
Please anybody correct me if I am thinking wrongly.
Yes, this zpool:

Code:
[17:22 r730-01 dvl ~] % zpool status zroot
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:33 with 0 errors on Thu Jul 16 04:02:31 2026
config:

    NAME                               STATE     READ WRITE CKSUM
    zroot                              ONLINE       0     0     0
      mirror-0                         ONLINE       0     0     0
        gpt/zfs0_20170718AA0000185556  ONLINE       0     0     0
        gpt/zfs1_20170719AA1178164201  ONLINE       0     0     0

errors: No known data errors
[17:22 r730-01 dvl ~] %

But the EFI partitions are not part of a zpool

Code:
=>       40  242255584  ada0  GPT  (116G)
         40       2008        - free -  (1004K)
       2048     409600     1  efi0_20170718AA0000185556  (200M)
     411648   16777216     2  swap0  (8.0G)
   17188864  225066760     3  zfs0_20170718AA0000185556  (107G)

[17:24 r730-01 dvl ~] %

If one of that pair dies, you've got the other EFI partition to boot off. That's my understanding; I'm not saying it's true/works.
 
Well, if nda4 is not that one is booting , you can ignore it, even in shoot-me mode, just answer 'No'. But, why it has a freebsd-boot partition?

My utility, in show-me mode, just tells where are the loaders and if there are up-to-date compared to those in /boot. It considers more things than you think.
That said, well, if you don't trust it, that's fine. Do it with hand, it works well also, until you make an error.

By the way, you need sometimes to be root to mount a partition; And in all cases, you need to be root to change the loaders.
 
Back
Top