Loader not found after upgrading the BIOS Boot Loader (Root on ZFS, freebsd-boot on USB)

M

_max_

I have two systems running for many years now. Upgrading the boot loader (via gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0) when upgrading from 14.3-RELEASE to 14.4-RELEASE (and even tested 15.1-RELEASE on one machine) left me with a system that does not boot without intervention. The loader on the root pool isn't found and I have to manually type zfs:zroot/ROOT/14.4-RELEASE:/boot/loader to get to the loader and boot the system.
If I remember it correctly I had the same issue on one of the earlier major upgrades and therefore left the old boot loader in place and stopped to upgrade the zfs pool. Sadly I now already upgraded the root pool on one of my machines.

Some more about my setup:
Code: 
$ sysctl machdep.bootmethod
machdep.bootmethod: BIOS
$ zpool list -o name,bootfs
NAME   BOOTFS
zroot  zroot/ROOT/14.4-RELEASE
$ gpart show
=>        40  3907029088  ada0  GPT  (2T)
          40  3907029088     1  freebsd-zfs  (2T)

=>        40  3907029088  ada1  GPT  (2T)
          40  3907029088     1  freebsd-zfs  (2T)

=>        40  3907029088  ada4  GPT  (2T)
          40  3907029088     1  freebsd-zfs  (2T)

=>        40  3907029088  ada2  GPT  (2T)
          40  3907029088     1  freebsd-zfs  (2T)

=>        40  3907029088  ada3  GPT  (2T)
          40  3907029088     1  freebsd-zfs  (2T)

=>     40  7679920  da0  GPT  (4G)
       40       16       - free -  (8K)
       56     1024    1  freebsd-boot  (512K)
     1080  7678880       - free -  (4G)

Maybe anybody can see what I did wrong? Already tried to install the bootcode from the running os or a usb live system without success. I even upgraded one machine to 15.1 and used the 15.1 pmbr und gptzfsboot.
 
Boot is on da0. ada[0-4]p1 are part of the zfs pool zroot.

For the exact messages before the boot prompt I am going to film the screen and copy them - but that might take a few days before I've got the time for that.
 
If it worked with a previous version of gptzfsboot, it sounds like a regression in its code.

Facts are BIOS booting is an ancient technology and will be less and less tested, furthermore like you use it, with an USB key; you might be the only one or so.

If you don't provide all the informations and tests, it's likely no FreeBSD dev will spend time at your possible future Problem Report.

This rise a question: can you try with UEFI? Because if your machine is UEFI capable, there are some chances you can workaround the problem like this.
 
What boot looks like:
Code: 
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
BIOS drive H: is disk5
BIOS drive I: is disk6
\
Can't find /boot/zfsloader

Can't find /boot/loader

FreeBSD/x86 boot
Default: disk-1:/boot/kernel/kernel
boot:

status shows the pool with the correct bootfs
zfs:zroot/ROOT/14.4-RELEASE:/boot/loader gives the loader and then the os

Yes, BIOS booting is ancient. But one of the machines isn't UEFI capable. Yes, I tried UEFI on the other one. Removed the freebsd-boot slice, replaced it with a 200M efi. But this also failed:
Code: 
Consoles: EFI Console
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk1p1:
FreeBSD/amd64 EFI loader, Revision 3.0

  Command line arguments: loader.efi
  EFI base: 0x735d6000
  EFI version: 2.40
  EFI Firmware: American Megatrends (rev 5.11)
  Console: efi (0x20000000)
  Load Path: \EFI\BOOT\BOOTX64.EFI
  Load Device: PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x4,0x0)/HD(1,GPT,4C4BEE6F-FC0C- <... screen cut on video>
,0x38,0x64000)
  BootCurrent: 0009
  BootOrder: 0007 0008 0009[*] 0003 0006
  BootInfo Path: VenHW(.....)
Ignoring Boot0009: No Media Path
Trying ESP: PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x4,0x0)/HD(1,GPT,4C4<...>
,0x38,0x64000)
Setting currdev to disk1pi:
Failed to find bootable partition
<last line cut on video>
 
Code: 
$ cat /boot/loader.conf
zfs_load="YES"
vfs.zfs.min_auto_ashift=12
kern.geom.label.disk_ident.enable=0
kern.geom.label.gptid.enable=0
kern.geom.label.gpt.enable=1

cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
coretemp_load="YES"

ipmi_load="YES"

kern.racct.enable=1

accf_http_load="YES"

#cuse_load=YES

# POWER OFF DEVICES WITHOUT ATTACHED DRIVER
hw.pci.do_power_nodriver=3
$ cat /boot/loader.conf.local
cat: /boot/loader.conf.local: No such file or directory
$ ls /boot/loader.conf.d/
$
 
From multiple testings in the past and current (in virtual machines), I observed that the FreeBSD loader (gptboot(8), gptzfsboot(8) and loader.efi(8)) boots the system only if the loader is on the same disk as the system is (BIOS and UEFI), contrary to the gptzfsboot(8) manual, which reads:
Rich (BB code): 
BOOTING
     gptzfsboot tries to find all ZFS pools that are composed of BIOS-visible
     hard disks or partitions on them.  gptzfsboot looks for ZFS device labels
     on all visible disks and in discovered supported partitions for all
     supported partition scheme types.  The search starts with the disk from
     which gptzfsboot itself was loaded.  Other disks are probed in BIOS
     defined order. After a disk is probed and gptzfsboot determines that the
     whole disk is not a ZFS pool member, the individual partitions are probed
     in their partition table order. Currently GPT and MBR partition schemes
     are supported.  With the GPT scheme, only partitions of type freebsd-zfs
     are probed.  The first pool seen during probing is used as a default boot
     pool.
Pity, there is no free space left on the system discs ada[0-4].

The only scenario I could boot from other disks while the gptzfsboot(8) loader was on a dedicated disc on a BIOS system, was to create a (UFS) partition on the dedicated disc and copy the systems /boot into it.

In addition setting in /boot/loader.conf of dedicated loader disc:
Code: 
vfs.root.mountfrom="zfs:zroot/ROOT/14.4-RELEASE"


_max_ said:
I tried UEFI on the other one. Removed the freebsd-boot slice, replaced it with a 200M efi. But this also failed:
Click to expand...
Same here on UEFI systems. If loader.efi(8) in the ESP (EFI system partition) is not on the same disk as the system is, it won't find a bootable system.

But you can specify the default device to loader the kernel from in a loader environment.

Assuming the ESP is mounted on /boot/efi/: /boot/efi/efi/freebsd/loader.env. Make sure there is a colon at the end of the variables value.
Code: 
rootdev=zfs:zroot/ROOT/14.4-RELEASE:
See loader.efi(8) for "loader.env". See loader_simp(8) for "rootdev".

Note loader.efi(8) reading loader.env in your UEFI systems boot screen print:
_max_ said:
Code: 
Consoles: EFI Console
    Reading loader env vars from /efi/freebsd/loader.env
Click to expand...


On the >=15 branches, the root file system variable can also be specified in a UEFI boot entry .

Example (assuming ada0p1 is the ESP):
Code: 
efibootmgr -c -a -L FreeBSD151 -l ada0p1:/efi/freebsd/loader.efi  -e "rootdev=zfs:zroot/ROOT/15.1-RELEASE:"
See efibootmgr(8) manual for options.
 
I see nothing wrong there. I'm surprised that the efi boot doesn't work too.
I noticed the gptzfsboot disk enumeration is counting 7 whereas you seem to have 6 disks.

I'm a little out of idea. What is the result of zpool status?

I ain't sure it' a good idea to set vfs.root.mountfrom with a BE. It's the job of bootfs property. What will happen if another BE is set as active?
 
Emrion said:
I noticed the gptzfsboot disk enumeration is counting 7 whereas you seem to have 6 disks.
Click to expand...
Sorry for the confusion. One machine has a hot-spare, while the other one used it's hot spare recently. Both have a usb drive for freebsd-boot and a 5 disk raidz pool zroot (+/- hot-spare). Wanted to rebuild the pools as 6 disk raidz2 for quite some time, but never found the time for that (and need a big drive to move the data off the old pool).

Emrion said:
I ain't sure it' a good idea to set vfs.root.mountfrom with a BE. It's the job of bootfs property. What will happen if another BE is set as active?
Click to expand...
Agree on that.

T-Daemon said:
From multiple testings in the past and current (in virtual machines), I observed that the FreeBSD loader (gptboot(8), gptzfsboot(8) and loader.efi(8)) boots the system only if the loader is on the same disk as the system is (BIOS and UEFI), contrary to the gptzfsboot(8) manual
Click to expand...
It worked in older versions. And as you stated, reading the documentation my setup should find the loader and boot. So either the boot loader can be fixed, or I get a drive and find the time to finally rebuild my pools and add a slice for freebsd-boot (or efi) on each drive.
Sadly I do not remember which version the working boot loader was from, but I still have an img of the slice, if that helps.
 
Reading the code of gptzfsboot, I see it tries to load these files in this order:
/boot/zfsloader
/boot/loader
/boot/kernel/kernel

If I understand correctly the code, it probes all the disks one by one, starting by the boot disk. It begins to seek a pool in the whole disk and if this doesn't work, it probes each partition it finds (libsa/zfs.c/zfs_probe_dev()).

So, it's exactly what T-Daemon reported with gtpzfsboot(8).

Code: 
printf("\nFreeBSD/x86 boot\n");
printf("Default: %s%s\nboot: ", boot_devname, kname);
So, disk-1: is boot_devname and /boot/kernel/kernel is kname, the last file tried.
The only conclusion possible is that it doesn't recognize your pool. Could it be because they have not been upgraded for too long?

You forgot zpool status.
 
One pool isn't upgraded for quite a while. The other one is upgraded and therefore does not boot with the old boot loader anymore (unsupported features). The pool is recognized by the boot loader:
_max_ said:
status shows the pool with the correct bootfs
Click to expand...

boot: status lists the pool with the correct bootfs

Code: 
$ zpool status

  pool: zroot
 state: ONLINE
  scan: resilvered 1010G in 21:03:18 with 0 errors on Tue Jun 30 11:09:11 2026
config:

        NAME              STATE     READ WRITE CKSUM
        zroot             ONLINE       0     0     0
          raidz1-0        ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0  
            gpt/<serial#> ONLINE       0     0     0  
            gpt/<serial#> ONLINE       0     0     0  
            gpt/<serial#> ONLINE       0     0     0  
            gpt/<serial#> ONLINE       0     0     0  

errors: No known data errors

Code: 
$ zpool status

  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 02:32:15 with 0 errors on Sat Jul  4 05:34:45 2026
config:

        NAME              STATE     READ WRITE CKSUM
        zroot             ONLINE       0     0     0
          raidz1-0        ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0
            gpt/<serial#> ONLINE       0     0     0
        spares
          gpt/<serial#>   AVAIL   

errors: No known data errors
 
_max_ said:
One pool isn't upgraded for quite a while. The other one is upgraded and therefore does not boot with the old boot loader anymore (unsupported features). The pool is recognized by the boot loader:
Click to expand...

What I mean is to test an upgraded pool with a recent bootloader. You can use the already upgraded pool, save the current gptzfsboot content (you dd the whole partition, pmbr is of no importance), update gptzfsboot and see what happen. I think, it's the less risky test.

You can also upgrade the other pool (the boot disk has a recent loader if I well understood), but before make a checkpoint.

See zpool-checkpoint(8). You can return to the non upgraded state after your test. But beware that doing that, it's like you rollback a snapshot. It's to do systematically before to upgrade a pool in case something goes wrong (but that's another subject).
 
Reading your efi booting report, it makes me think that the loader tries only on the boot disk. Too bad you didn't have the last lines.

What you may (also) try is to create a file /efi/freebsd/loader.env in the ESP which contains only: rootdev=disk2p1

Edit: Ok, I tried on a VM (14.2-RELEASE) and I found no way to boot if it's not the same disk that has both loader and root system, BIOS or EFI booting. T-Daemon was right on this point.

So, it's a huge regression because it concerns BIOS and EFI loaders. I'm surprised that no one complained about this. Furthermore, the documentation states the opposite for gptzfsboot and the code seems to do that but it doesn't. A PR must be created.
 
There already is PR 296309. If there is anything to add, or if it deserves a complete new PR, I suggest to anyone who has more intracate knowledge and experience with this than me, to act accordingly. Also, when the commit mentioned by bsdimp is reversed (I expect first in -CURRENT and -STABLE), verify that it solves the problem of this thread.

Edit: given the date of the commit mentioned and the fact that the PR is reported for -CURRENT, a new new PR, relating to a -RELEASE seems more appropriate.
 
Emrion I wonder how often the specific situation actually happens. My personal experience (granted sample size of one over a number of years/releases) my loader and root FS were always on the same device/devices (for mirrors).
I think some configurations related to GELI are different with loader on one device and root FS (encrypted) on another.
So yes it may be a huge regression but it may be on a corner case which would lead to not reporting.

Regardless: yes, bug report get it fixed.
 
Erichans said:
There already is PR 296309. If there is anything to add, or if it deserves a complete new PR, I suggest to anyone who has more intracate knowledge and experience with this than me, to act accordingly. Also, when the commit mentioned by bsdimp is reversed (I expect first in -CURRENT and -STABLE), verify that it solves the problem of this thread.

Edit: given the date of the commit mentioned and the fact that the PR is reported for -CURRENT, a new new PR, relating to a -RELEASE seems more appropriate.
Click to expand...
Note that the PR (mine) is for UEFI boot and haven't test if the same applies to BIOS boots.
Another thing to mention is that I have patched (with my variant at PR 207940) boot1.efi as EFI/boot/bootx64.efi in ESP to kick /boot/loader.efi, which is irregular in recent installations. But the boot1.efi is NOT affected with or without the offending commit, which is NOT MFC'ed to any of stable branches.
So any releng branches, which are used for *-RELEASE{-p*} builds, doesn't have, too.
 
Thanks. I think the problem is older and different as it affects both gptzfsboot and loader.efi (it may be actually two not related bugs). I tested with a 14.2-RELEASE bootcodes, so it can't be a recent commit. As mer stated, few people boots on a disk that not also holds the root fs.

For the moment, I will try to see what I can do with gptzfsboot.
 
Emrion said:
As mer stated, few people boots on a disk that not also holds the root fs.
Click to expand...
That may, or may not be the case; fact is that, as reported here, it breaks two -RELEASE versions.
It may also be a lingering error, just waiting to come to the surface, because I think a lot of users may not be in a hurry to update their bootloaders: they can get by just fine given that they do not update their ZFS boot pool. It also seems that users booting UFS are not affected.

P.S. Today a number of commits to -CURRENT wrt the loader (2026-07-10 - main) were made, including the mentioned announced reversal: Revert "loader.efi: Trim ZFS searching for other booting options"
 
I still have the old loader (the last one working for me) on img. Maybe it helps to find out which FreeBSD version it came from? Already tried strings da0p1.img but I found nothing about a FreeBSD version.
 
_max_ said:
I still have the old loader (the last one working for me) on img. Maybe it helps to find out which FreeBSD version it came from? Already tried strings da0p1.img but I found nothing about a FreeBSD version.
Click to expand...
You did zpool upgrade on one of the systems already, but not on the other, yes?
If so, the old loader would help on one system but not the other, so maybe a fixed new loader.
As Erichans notes, a bunch of commits went into -CURRENT so maybe an image from there?
 
Sorry, I think I didn't make myself clear. I meant if my images or info about what version the last working loader came from can help anybody narrowing down the problem?
 
You state you went from 14.3 to 14.4; did the system start at 14.3 or was it upgraded from say 14.1 to 14.3?
That information may help figure it out.
You could try putting the old .img file on a new USB device and booting that
 
As you already said that you've upgraded the root pool (should mean zpool upgrade), you basically cannot use older version of boot codes.

This is because boot code that are kicked by (P)MBR need to know how to read the boot pool.

And upgrading pool usually mean that some read-incompatible features are enabled, thus, cannot be read by older boot codes that doesn't know how to read from pools that any of read-incompatible feature is actually used.

And as far as I know, boot codes are maintained NOT to drop supports for read-incompatible features of ZFS pools even if it mean some other features to be dropped.

Note that boot codes that runs prior to loader for legacyb BIOS boots mutually has quite strict restrictions in size.

You can see how strict boot codes in MBR is here.

And as partition boot records at the top of the physical partitions (for MBR) or dedicated freebsd-boot partition (for GPT) is forced to run under real mode of the x86{-64} CPU, it's restricted to 540kiB to avoid memory areas for VRAMs, BIOS ROMs and memory mapped I/Os.

I believe ZFS boots via legacy BIOS (or worse, legacy BIOS boot itself) would be dropped at some point in the furure. Otherwise, ZFS on FreeBSD is restricted to disallow upgrading ZFS pools (precisely, pools including bootfs, but possibly including others to be safest).
 
You must log in or register to reply here.
Back
Top