Loader not found after upgrading the BIOS Boot Loader (Root on ZFS, freebsd-boot on USB)

[Emrion]

Sorry, I think I didn't make myself clear. I meant if my images or info about what version the last working loader came from can help anybody narrowing down the problem?

As far as I know, there is no FreeBSD version in gptzfsboot. So, I don't think it will help.
It should come from the 13 era, if I well understood your story.

Reading the code of gptzfsboot releng/15.1, I already have an idea of what went wrong, but it's just an intuition. I need to make some tests.

 

[Emrion]

And as partition boot records at the top of the physical partitions (for MBR) or dedicated freebsd-boot partition (for GPT) is forced to run under real mode of the x86{-64} CPU, it's restricted to 540kiB to avoid memory areas for VRAMs, BIOS ROMs and memory mapped I/Os.

pmbr runs in real mode, but gptzfsboot switches to virtual mode thank to the BTX loader that runs before the main code.
So, no there is no technical reason I know to remove BIOS support. It will happen, but I think more for lack of interest and dev people.

 

[Erichans]

Just to be clear, if you install FreeBSD on a seperate (USB) disk (ZFS or non ZFS) that has a "new" bootloader and (minimal) bootable OS system itself*, you should be able to import the RAIDZ pool (full ZFS support is then available through the ZFS kernel module). For a UFS based system you'll have to load ZFS as "an extra" to be able to import your RAIDZ ZFS pool.

(and even tested 15.1-RELEASE on one machine) left me with a system that does not boot without intervention. The loader on the root pool isn't found and I have to manually type zfs:zroot/ROOT/14.4-RELEASE:/boot/loader to get to the loader and boot the system.
If I remember it correctly I had the same issue on one of the earlier major upgrades and therefore left the old boot loader in place and stopped to upgrade the zfs pool. Sadly I now already upgraded the root pool on one of my machines.

Note however, if you've upgraded your RAIDZ pool when running 15.1-RELEASE, you'll likely have to install 15.1-RELEASE on the seperate USB disk as well (not 14.4-RELEASE), as FreeBSD 15.1-RELEASE has ZFS version 2.4.2 and 14.4-RELEASE has ZFS version 2.2.9.

___
* running an OS from a USB device, especially a USB stick, wouldn't be my first choice obviously but, at least you'd have access to your RAIDZ pool.

 

[T-Aoki]

pmbr runs in real mode, but gptzfsboot switches to virtual mode thank to the BTX loader that runs before the main code.
So, no there is no technical reason I know to remove BIOS support. It will happen, but I think more for lack of interest and dev people.

Precisely, start on real mode and switch to 32-bit protected mode by contained btx server to allow loading larger /boot/loader. So, this stage of boot codes are read while in real mode, thus, need to fit into real mode memory space.
If you're interested and want quite detailed (than source codes itself) information, you can read Chapter 1 of FreeBSD Architecture Handbook.

 

[Emrion]

Precisely, start on real mode and switch to 32-bit protected mode by contained btx server to allow loading larger /boot/loader. So, this stage of boot codes are read while in real mode, thus, need to fit into real mode memory space.
If you're interested and want quite detailed (than source codes itself) information, you can read Chapter 1 of FreeBSD Architecture Handbook.

Thank, but I know this page (it doesn't really speak of pmbr nor gptzfsboot, but the booting with a mbr scheme). My point was to said that memory isn't a limit for BIOS loader like gptzfsboot. It seems to me that you stated it runs in real mode. Is it not what I quoted?

Ok, I get it. The code is loaded in real mode. But, this code can access to a vast portion of memory once it runs. Nothing prohibits to load some more code from the partition itself (need to modify pmbr), or another one, for example.

 

[T-Aoki]

Thank, but I know this page. My point was to said that memory isn't a limit for BIOS loader like gptzfsboot. It seems to me that you stated it runs in real mode. Is it not what I quoted?

As written in the page, first sector (512bytes) of partition boot codes (or boot codes in freebsd-boot partition for GPT) alone is loaded by MBR in real mode and there's boot1 only.

Remaining part (actually, whole part) are loaded by boot1 still in real mode.
Roughly, boot1 passes control to loaded boot2 (just after the boot1 area, 7680bytes) and it includes btx which switches to 32bit protected mode.

Possibly I'm mis-understanding, but the transition to 32bit happenes AFTER boot2 looks for /boot/loader and load it into memory, thus, /boot/boot (concatenated file of /boot/boot1 and /boot/boot2) or /boot/zfsboot (for ZFS) with /boot/loader needs to fit into real mode memory area.

If I understand correctly, the transition happenes after loading /boot/loader in boot2 (should be included in /boot/zfsboot, too). For amd64, transition to 64bit is done in /boot/loader.

On the other hand, things are far more simpler on UEFI.
This is because UEFI firmware itself boots on the mode (32bit or 64bit protected mode) it is built for. And havind far less restriction (need consideration for minimum supported environments).

This document by Klara would be helpful, too.

 

[Emrion]

Once again my intuition was right.

I looked into stand/i386/gptzfsboot/zfsboot.c from releng/15.1, function i386_zfs_probe:

for (dev.dd.d_unit = 0; bd_unit2bios(&dev) >= 0; dev.dd.d_unit++) {
        snprintf(devname, sizeof (devname), "%s%d:", bioshd.dv_name,
            dev.dd.d_unit);

        /* If this is not boot disk, use generic probe. */
        if (dev.dd.d_unit != boot_unit)
            zfs_probe_dev(devname, NULL, true);
        else
            zfs_probe_dev(devname, &pool_guid, true);

        if (pool_guid != 0 && bdev == NULL) {
            bdev = malloc(sizeof (struct i386_devdesc));
            bzero(bdev, sizeof (struct i386_devdesc));
            bdev->zfs.dd.d_dev = &zfs_dev;
            bdev->zfs.pool_guid = pool_guid;
        }
    }

What does that means "If this is not boot disk, use generic probe."?
So, if the disk is not the boot disk, it passes NULL in lieu of pool_guid.
Generic probe ok, but how can it works without this pool_guid?
It's obvious that without this var, bdev isn't filled, even if a pool is found. And that's exactly the case.

So, I just got rid of this test and always call zfs_probe_dev(devname, &pool_guid, true);. Now, gptzfsboot can boot another disk than the booting one.

I think this is more a workaround than a solution to the bug, because I suspect the guy that added these lines did that for a good reason. Another point is that doesn't explain why loader.efi is also impacted. It uses, as gptzfsboot, libsa, so it's maybe a bug in libsa in fact. That said, I found nothing weird on this way.

I will fill a PR, but I want to read more loader.efi and libsa code before, maybe the light will come. In the meaning time, I post a gptzfsboot that should work for _max_.

 

[T-Aoki]

Once again my intuition was right.

I looked into stand/i386/gptzfsboot/zfsboot.c from releng/15.1, function i386_zfs_probe:

for (dev.dd.d_unit = 0; bd_unit2bios(&dev) >= 0; dev.dd.d_unit++) {
        snprintf(devname, sizeof (devname), "%s%d:", bioshd.dv_name,
            dev.dd.d_unit);

        /* If this is not boot disk, use generic probe. */
        if (dev.dd.d_unit != boot_unit)
            zfs_probe_dev(devname, NULL, true);
        else
            zfs_probe_dev(devname, &pool_guid, true);

        if (pool_guid != 0 && bdev == NULL) {
            bdev = malloc(sizeof (struct i386_devdesc));
            bzero(bdev, sizeof (struct i386_devdesc));
            bdev->zfs.dd.d_dev = &zfs_dev;
            bdev->zfs.pool_guid = pool_guid;
        }
    }

What does that means "If this is not boot disk, use generic probe."?
So, if the disk is not the boot disk, it passes NULL in lieu of pool_guid.
Generic probe ok, but how can it works without this pool_guid?
It's obvious that without this var, bdev isn't filled, even if a pool is found. And that's exactly the case.

So, I just got rid of this test and always call zfs_probe_dev(devname, &pool_guid, true);. Now, gptzfsboot can boot another disk than the booting one.

I think this is more a workaround than a solution to the bug, because I suspect the guy that added these lines did that for a good reason. Another point is that doesn't explain why loader.efi is also impacted. It uses, as gptzfsboot, libsa, so it's maybe a bug in libsa in fact. That said, I found nothing weird on this way.

I will fill a PR, but I want to read more loader.efi and libsa code before, maybe the light will come. In the meaning time, I post a gptzfsboot that should work for _max_.

For now, just one thing to reply.

Using "blame" link in cgit on reading the file would help tracking changes.
In this case, https://cgit.freebsd.org/src/blame/stand/i386/gptzfsboot/zfsboot.c?h=releng/15.1.
At the leftmost part, by which commit the block is last modified can be seen.
And clicking the commit hash there navigates you to the commit and can read commit messages and diffs.
By clicking "root" on path: line near the top would show you all modified / added / deleted files by the commit.