Why it's important to upgrade /efi/boot/BOOTx64.efi

E

Emrion

I noticed last week on a VM (FreeBSD as guest on VirtualBox, efi boot) that the boot environment didn't work when you select it on the beasty menu. The kernel runs until it comes to mount root. Then, you read "internal error: failed to initialize ZFS library". After that, it seems that the ZFS root filesystem remains read-only. You can imagine easily that it crashes soon after because there is no device in /dev for example.

But, if you activate a BE with beadm (for instance) and reboot, all is working well.
I even experimented no boot at all with efi but cannot tell the exact circonstances.

I eventually understood where the problem lies. To tell the story briefly, I came from FreeBSD 12.0 (or before) and never updated this essential efi file, I mean /efi/boot/BOOTx64.efi.

My previous BOOTx64. efi: 393216 bytes.
The 12.1 loader.efi: 460800 bytes (no problem with 12.2).
The 12.2 loader.efi: 488960 bytes.

So, at each OS upgrade, even minor, do something like that:
mount -t msdosfs /dev/drive-name-partition /mnt
cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi

drive-name-partition is often ada0p1 but you have to look where the efi partition lies with gpart show.
 
In a VM, I did a quick 12.2 install. I then upgraded with freebsd-update. (Note that this was just a single ZFS virtual drive, not a mirror or other, more sophisticated setup.) It was able to boot. I also ran zpool upgrade -a as 13 is using ZFS 2. I had wondered, due to various posts I've seen here as well as some information in release notes, if this would work without me doing anything to the efi loader.
www.freebsd.org

FreeBSD 13.0-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

So far, on one VM with EFI boot (in the VM--VirtualBox to be specific, running on CentOS 8.3 stream, version 6.1.18, and another test, on bare metal, but again a single drive laptop, upgrading from 12.2 to 13.0 and then upgrading the zpool has given me no problems. These are very simple setups, just for testing, but note that 13.0 release notes also mention the boot loader on efi. (See link above).
 
The problem I was talking about is very vicious. I think I kept the efi loader of the 12.0-RELEASE which worked perfectly on 12.1 but not on 12.2. And you see the problem only if you try to activate a BE with the beasty menu (thing that most people don't do often).

I tried to understand the trouble with the installation of a 12.1-RELEASE in a VM. Updated it and then, upgraded to 12.2 (without changing the efi loader). I made several BEs before upgrading zpool and after. All was working, no problem (including starting from the beasty menu)... Why? Because the efi loader of the 12.1-RELEASE works on the 12.2.

So, in this case, the efi loader is compatible with the version+1 but not with version+2. That means if you don't update this loader at each upgrade, something will bite you later.

The problem is also due to that freebsd-update tells you to update the freebsd-boot partition even if your boot method is UEFI and even if you don't have one. But, in no way, it talks about the update of BOOTx64.efi.
 
Emrion said:
The problem I was talking about is very vicious. I think I kept the efi loader of the 12.0-RELEASE which worked perfectly on 12.1 but not on 12.2. And you see the problem only if you try to activate a BE with the beasty menu (thing that most people don't do often).
Click to expand...
And you can prevent this failure by upgrading the loader first. This desktop, I am writing this message on, has 12.2, but I have already upgraded the EFI loader to 13.0-RC3. This is backward compatible. And the file size is 896000. Almost twice as big and needs a bigger partition if you still have an old 800K EFI partition.

It works very well with 12.2 and one day I decide to upgrade this machine, I am not struggling with boot issues.

BTW, the loader can be built independently of base and kernel by:
Code: 
cd /usr/src/stand/
make
 
Argentum said:
This is backward compatible.
Click to expand...
Is this guarantee? I mean, is it documented somewhere or just the result of your experiments?
Will the 14.0 efi loader work on a 13.x system?

I said that because I had a problem with the gptzfsboot of a 12.0-RELEASE which was incompatible with the 11.2-RELEASE: https://forums.freebsd.org/threads/cant-load-kernel-when-back-to-be-11-2-from-12-0-release.68795/

Normally, you just have to update the efi loader when freebsd-update advise you to update the freebsd-boot partition (which you should do as well if you have the said partition type).
 
Emrion said:
I said that because I had a problem with the gptzfsboot of a 12.0-RELEASE which was incompatible with the 11.2-RELEASE
Click to expand...
11.2 is newer than 12.0. Counter-intuitive I know but a result of the way new versions are released.
 
SirDice said:
11.2 is newer than 12.0. Counter-intuitive I know but a result of the way new versions are released.
Click to expand...

It seems that this is the case of 11.3−RELEASE but not 11.2. Am I wrong?
Click to expand...
 
Emrion said:
Is this guarantee? I mean, is it documented somewhere or just the result of your experiments?
Will the 14.0 efi loader work on a 13.x system?
Click to expand...
A good question. Making predictions is a hard job, especially about future ;)

But, as long as ZFS is in scope, I think it will be backward compatible. I dug a bit inside the 12.2 loader, rebuilt it with my ownd debug messages, to find out why it did not boot the upgraded OpeZFS pool. And the reason is that loader has a simple stand-alone implementation of ZFS and old implementation does not recognize new pool version. The general policy of ZFS development has been backward compatibility. So, it was not a surprise that new (13.0) loader can boot old system, but old loader in general may not boot new system.
 
I do recall it's been said numerous times, that you should only upgrade the zfs pool after you know for certain that you are not going to use it on a older system. It ends up being, it does keep backwards compatibility, there is nothing saying it will keep forward compatability of which it doesn't know anything of (future changes).
 
Looks like you can recover from a zpool upgrade now by setting a "checkpoint" before you do it, as the loader is able to rewind it?
www.oshogbo.vexillium.org

Bringing zpool checkpoints to a FreeBSD bootloader

Almost two years ago I wrote a blog post about checkpoints in ZFS. I didn’t hide that I was a big fan of them. That said, after those two years, I still feel that there are underappreciated features in the ZFS world, so I decided to do something about that.
www.oshogbo.vexillium.org www.oshogbo.vexillium.org

Of course, it's still wise not to do it early, rewinding will erase anything done to the pool after the checkpoint…
 
ct85711 said:
I do recall it's been said numerous times, that you should only upgrade the zfs pool after you know for certain that you are not going to use it on a older system. It ends up being, it does keep backwards compatibility, there is nothing saying it will keep forward compatability of which it doesn't know anything of (future changes).
Click to expand...
This was my test system, and I was trying to get the OpenZFS pool booting before 13.0 was out. That was the intention of the experiment.

To get the updated pool back, the easiest and foolproof way is to create an empty lower version pool and using zfs send and zfs receive move the upgraded pool to the empty lower version pool. Set it bootable and everything is OK. That is what I did with my test system.
 
I always make a checkpoint before upgrading a pool, by principle. But a checkpoint is something you can't keep for too long because it prohibits some zfs functionalities.

Anyway, as I said previously, I experimented a case where zfs backward compatibility wasn't, at least for BIOS booting: 11.2 --> 12.0. So, i won't count on any backward or forward compatibility and, except for testing purposes, no one should.
 
I think it would be great to include a mention of when the boot loader needs to be updated. Every time after the OS upgrade? Is updating boot loader after major version upgrades only acceptable? Also, a little bit of guidance on when the boot loader must be updated along the upgrade process would really help, otherwise it's somewhat unclear. Do I do it before the first reboot?
 
blackhaz said:
Do I do it before the first reboot?
Click to expand...
For ZFS systems, you should do it before upgrading the pool you boot from. If you upgrade the pool but didn't update the boot code you can end up with a non-booting system.
 
blackhaz said:
Do I do it before the first reboot?
Click to expand...
No. You must finish up the system upgrade (you need a reboot for the kernel at least).

This becomes mandatory when you upgrade the pool ( zpool upgrade).

All the update of bootcodes is unclear in FreeBSD. Worse, it evolves a few from a major release to the other. Depending on the version you installed, it's not /efi/boot/bootx64.efi which boots anymore but /efi/freebsd/loader.efi.

You can see it with efibootmgr -v.
 
It would be great to have a reference to §22.3.11 in Chapter 26 somewhere - perhaps a step-by-step upgrading guide.
 
grahamperrin said:
I used to boot without anything detectable by efibootmgr(8).
Click to expand...
Because the default is /efi/boot/bootx64.efi if no path is specified (x64 based system). But, if you mean you see no entry at all (no boot0000 and so on), I've never seen a machine in this case.

Edit: in a FreeBSD VM, I deleted all the boot entries and, yes, it started anyway. In fact, the efi BIOS reconstructed the boot entries. The ones you have by default on VirtualBox. In this case: UiApp, UEFI VBOX CDROM, UEFI VBOX HARDDISK (booted on this one) and EFI Internal Shell.
 
Sorry,
Emrion said:
… if you mean you see no entry at all …
Click to expand...

My memory wasn't quite right. The example under <https://www.freebsd.org/releases/14.0R/relnotes/#upgrade>:

Code: 
+Boot0000* FreeBSD HD(1,GPT,f859c46d-19ee-4e40-8975-3ad1ab00ac09,0x800,0x82000)/File(\EFI\freebsd\loader.efi)
                      nda0p1:/EFI/freebsd/loader.efi (null)

– is very different from what I got in response to the same command:

Code: 
Boot to FW : false
BootCurrent: 0000
Timeout    : 0 seconds

<https://codeberg.org/grahamperrin/freebsd-doc/pulls/38#issuecomment-1287230>
 
Emrion said:
I noticed last week on a VM (FreeBSD as guest on VirtualBox, efi boot) that the boot environment didn't work when you select it on the beasty menu. The kernel runs until it comes to mount root. Then, you read "internal error: failed to initialize ZFS library". After that, it seems that the ZFS root filesystem remains read-only. You can imagine easily that it crashes soon after because there is no device in /dev for example.

But, if you activate a BE with beadm (for instance) and reboot, all is working well.
I even experimented no boot at all with efi but cannot tell the exact circonstances.

I eventually understood where the problem lies. To tell the story briefly, I came from FreeBSD 12.0 (or before) and never updated this essential efi file, I mean /efi/boot/BOOTx64.efi.

My previous BOOTx64. efi: 393216 bytes.
The 12.1 loader.efi: 460800 bytes (no problem with 12.2).
The 12.2 loader.efi: 488960 bytes.

So, at each OS upgrade, even minor, do something like that:
mount -t msdosfs /dev/drive-name-partition /mnt
cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi

drive-name-partition is often ada0p1 but you have to look where the efi partition lies with gpart show.
Click to expand...
If I only had read this a couple of weeks before... =D
 
grahamperrin said:
For predictability and consistency, probably better to make the mount point:

/boot/efi
Click to expand...
Before 13.x, the ESP partition wasn't automatically mounted. I mean a fresh installation of 13.x, because if you upgraded from a 12.x version, nothing happened on this side (and /boot/efi remains unmounted). I think the 13.2-RELEASE installed /efi/freebsd/loader.efi and used efibootmgr to select this last for boot, leaving /efi/boot/bootx64.efi unused (second in boot order).

I think that to leave the ESP partition mounted implies a risk. But... If one day, FreeBSD devs choose to automatically update the efi loader at each upgrade, I will take that risk. From my point of view, the path is taken: with a FreeBSD dedicated loader (/efi/freebsd/loader.efi), there is no or less problems to update that loader.

So, yes, to see this miracle happen, it's better to have /boot/efi mounted.
 
You must log in or register to reply here.
Back
Top