Utility that tries to figure how to update the FreeBSD loaders

loaders-update is intended for simple systems and, in some ways, inexperienced users.
Agreed.

Note that, as a fallback, it's simpler to keep an USB key with an appropriate version of FreeBSD inside. I mean if you have a physical access to the machine.

That said, I never experienced a boot problem after updating the loaders.
Unfortunately, I've experienced the situation before.
With the broken boot1.efi (at the moment, loader.efi must be kicked from boot1.efi), even when booted from USB memstick, FreeBSD which was in the first recognized hard drive by firmware was booted, thus cannot repair using memstick on UEFI boot.

But fortunately, at the moment, I had freebsd-boot partition for legacy boot (just using it when I started to test UEFI boot) and I could rescue myself to be able to boot with disabling UEFI boot on firmware config.

The top of thread on freebsd-current ML reporting the issue.

Review of the fix was D5108.
As I didn't registered (as I'm not sure I, not a FreeBSD committer is allowed to register there or not) at the moment, my reports are all via email (on or off list).
 
loaders-update is intended for simple systems and, in some ways, inexperienced users.

yes and no

It's really fast for an experienced users to forget a boot loader update after a zpool upgrade.

This check process part should be included in documentation and base script of freebsd-update.

Maybe it's already in.

But I my case, I miss the boot loader upgrade one or two time after a pool upgrade and it's not a fun case.
 
yes and no

It's really fast for an experienced users to forget a boot loader update after a zpool upgrade.

This check process part should be included in documentation and base script of freebsd-update.

Maybe it's already in.

But I my case, I miss the boot loader upgrade one or two time after a pool upgrade and it's not a fun case.
True. I don't consider myself as inexperienced and I use this software on all my machines. It's quicker and avoid typing errors.

At the base, I wrote this because of the recurrent problems/questions/disasters that we get here from people, often new users, but not only.

Of course, I was bitten as well by this, but in an uncommon way: https://forums.freebsd.org/threads/why-its-important-to-upgrade-efi-boot-bootx64-efi.79503

I agree, this should be part of freebsd-update with all the precautions needed by this delicate operation. I ported this software also in this idea, to attract the attention of the base committers.

From my point of view, it's one of the two main problems in using/installing/discovering FreeBSD and certainly a reason why some people fled this OS. The other is the kernel modules that remain unusable with the package system during the first three months of a minor release. This is incomprehensible to a standard user. On that point, I have no solution so far, but I'm thinking about it.
 
It's really fast for an experienced users to forget a boot loader update after a zpool upgrade.
It should be BEFORE zpool upgrade.

It would be best (but not sure it could be accepted upstream) if zpool upgrade forcibly invoke boot code update if any of the specified pool have bootfs and error out without upgrading if it doesn't finished successfully.
 
Well, upgrading boot code AFTER upgrading pool could be hazardous.

Imagine if the computer crashes just after upgrading the pool is finished, but before updating boot codes which support the newly enabled features with some reason (sudden AC line failures etc.).
It can easily cause unbootable computer.

And this also applies if you manually enable/activate any of the features that currently installed boot code doesn't support.
This case needs some more investigations before doing. You need to look for newer releases/stables and main (aka current) which its boot codes support the feature. You need to update your boot code of which you found.

Note that read compatible features are safe. Only read incompatible features matter.
 
I was saying that I think the bootcodes update should be done at system upgrade time (during freebsd-update). Because some problems may arise even if you didn't upgrade the pool as in the link I posted. So, it's before upgrading any pool and even if you don't upgrade your pool(s).
 
I was saying that I think the bootcodes update should be done at system upgrade time (during freebsd-update).
100% agreed for binary upgrade (freebsd-update) users.
But not all users do binary upgrade but source upgrade (mainly developers and early adopters). So forcible upgrade on freebsd-update time is insufficient (as human makes mistake).

Because some problems may arise even if you didn't upgrade the pool as in the link I posted. So, it's before upgrading any pool and even if you don't upgrade your pool(s).
Do you mean that you couldn't sanely boot / mountroot on some BEs?
After loader hands over to kernel, bootcodes does not matter anymore.
zfs.ko loaded by loader (before kernel starts, via /boot/loader.conf[.local]) and ZFS-related userland components like /sbin/zfs, /sbin/zpool, /lib/libzfs* must be in sync, at least zfs.ko should not be older than others.
Otherwise, syscalls from userland could fail. There could be other reason, but this is what I can think of for now.

For sane and safe upgrade of boot codes, it would be desirable that what read incompatible features are supported by boot codes can easily confirmed.

Fortunately, /usr/src/stand/libsa/zfs/zfsimpl.c has an array named features_for_read, but unfortunately, for example, string /boot/loader.efi does not list the array as a bulk.
If a file containing the array as text and hashes of boot codes in /boot as, for example, /boot/zfsfeatures.list is created on building boot codes, it would be helpful.
 
But not all users do binary upgrade but source upgrade (mainly developers and early adopters).
They don't use freebsd-update, so no automatic bootcodes update. It makes sense.

Do you mean that you couldn't sanely boot / mountroot on some BEs?
It was what I experienced. After updating the bootcodes, no problem anymore. That said, I didn't put the analysis deeper.
 
Emrion : Great utility, kudos. I have an OPNSense system running ZFS that is working OK, but I would like to preemptively update the bootloader(s) to prevent future issues. Before I do, I would like to understand what the recovery method would be in case I were left with an unbootable system after running the bootloader update. What steps should I take to prepare for a possible failure? I have physical access to the unit and can use a serial console connection if required.
 
Emrion : Great utility, kudos. I have an OPNSense system running ZFS that is working OK, but I would like to preemptively update the bootloader(s) to prevent future issues. Before I do, I would like to understand what the recovery method would be in case I were left with an unbootable system after running the bootloader update. What steps should I take to prepare for a possible failure? I have physical access to the unit and can use a serial console connection if required.
Preemptively you say? Just after the upgrade of the system or before, meaning you get the bootloaders of the next version and put them in a directory (and use the -s option)? I don't advise this last.

Anyway, the simplest method is to have a USB key with the current version of your OS. And when it concerns something that seats between internet and your local network, create this key before attempting any upgrade ;). If a problem arises, you boot on this key with the BIOS menu. Then, you can downgrade the loaders if necessary.

I never faced any trouble with the systematic update of the loaders, but I did it always after upgrade (including reboot for the new kernel takes place). It's better to reboot again immediatly to be sure that all is ok.
 
Hi Emrion,
I have couple of cases where the outputs of loaders-update confuse me. Let me start with this one.
question 1, since the system is UEFI boot, why not run gpart bootcode -p /boot/boot1.efi -i 2 nda1, as handbook says, https://docs.freebsd.org/en/books/handbook/zfs/#zfs-zpool-upgrade, (scroll down a few screenshots, see the "Warning" section.)
question 2, because the output reports a serious error, which i suppose refers to /dev/nda2p2, is it still safe to run loaders-update shoot-me?
Code:
root@AMD_STATION:[~] # gpart show
=>        34  1953525101  nda0  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048      204800     1  efi  (100M)
      206848       32768     2  ms-reserved  (16M)
      239616  1131192599     3  ms-basic-data  (539G)
  1131432215        1769        - free -  (885K)
  1131433984     1736704     4  ms-recovery  (848M)
  1133170688        2048        - free -  (1.0M)
  1133172736   819200000     5  ms-basic-data  (391G)
  1952372736        2048        - free -  (1.0M)
  1952374784     1146880     6  ms-recovery  (560M)
  1953521664        3471        - free -  (1.7M)

=>        40  1953525095  nda1  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712        1423        - free -  (712K)

=>        34  2000409197  nda2  GPT  (954G)
          34           6        - free -  (3.0K)
          40      532480     1  efi  (260M)
      532520        1024     2  freebsd-boot  (512K)
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G)
     4728832  1948794880     4  freebsd-zfs  (929G)
  1953523712    46885519        - free -  (22G)

root@AMD_STATION:[~] # loaders-update show-me
loaders-update v1.2.2

One or more efi partition(s) have been found.

Examining nda2p1...
mount -t msdosfs /dev/nda2p1 /mnt
EFI loader /mnt/EFI/BOOT/BOOTX64.EFI is up-to-date.
Would run: cp /boot/loader.efi /mnt/EFI/BOOT/bootia32.efi
EFI loader /mnt/EFI/freebsd/loader.efi is up-to-date.
umount /mnt
Examining nda1p1...
Efi partition nda1p1 is already mounted in /boot/efi.
EFI loader /boot/efi/efi/freebsd/loader.efi is up-to-date.
Would run: cp /boot/loader.efi /boot/efi/efi/boot/bootia32.efi
EFI loader /boot/efi/efi/boot/bootx64.efi is up-to-date.
Examining nda0p1...
mount -t msdosfs /dev/nda0p1 /mnt
There is no FreeBSD loader in nda0p1
umount /mnt

One or more freebsd-boot partition(s) have been found.
The root file system is zfs.

Examining nda2...
There is currently no loader in /dev/nda2p2.
Cannot update without option -f.
Examining nda1...
Would run: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1

-------------------------------
Your current boot method is UEFI.
Boot device: /dev/nda1p1 File(\EFI\BOOT\BOOTX64.EFI)
Updatable EFI loader: 2
Updatable BIOS loader: 1
1 serious error(s) occured. See texts above.
 
(First of all, according to your partition layouts, I assume your nda0 to be of Windows, so completely ignore it.)

This should be because you have ESP as nda1p1 and freebsd-boot partition as nda1p2 (same for nda2, too).
ESP (uEfi System Partition) is not at all needed for legacy BIOS boots, while freebsd-boot is not at all needed for UEFI boots. Having both should be confusing you.

And more, ESP is mandated to be formatted as FAT{12|16|32}, while freebsd-boot partition is blindly read into memory as the whole partition to be memory image starting from fixed physical memory address by bootcode in (p)MBR.

Read Chapter 15 of the Handbok for details about legacy BIOS boots.
And this would help, too.
 
The method to update an EFI bootcode with gpart bootcode is not only outdated, but it will also destroy your ESP partition in a way you won't boot anymore (I think, not tested). So, stay away from that.

As T-Aoki says, you have an EFI partition that Windows uses in nda0p1. loaders-update saw it has no FreeBSD loader, therefore and fortunately, it won't do anything there. I programmed it for that: don't touch other loaders than the ones belonging to FreeBSD!

Concerning nda2p2, you don't have a bios bootcode inside. You can add one or not, that's not important since you boot on EFI. This is the thing that makes a "serious error".

Most of your EFI loaders are up-to-date.

What is worry me is that it wants to update two loaders named bootia32.efi. How these files came there? If I understand correctly, it's for 32-bits EFI. You have 64-bits EFI since your system boots on \EFI\BOOT\BOOTX64.EFI. If I was you, I'd delete these files.

Well, it's not really useful to run loaders-update shoot-me because it won't do anything gainful.
If you want to continue to use it without error or weird things, delete these bootia32.efi files and run it in shoot-me mode with the -f option. After that, it will be happy, saying all is up-to-date.
 
Thanks to T-Aoki and Emrion.
If I understand this right, in my case, gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1 writes to freebsd-boot partition.
while cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi changes efi partition. gpart bootcode -p /boot/boot1.efi -i 1 nda1 writes to efi partition as well, but as Emrion explains, I shouldn't use it. Don't know what gpart bootcode -p /boot/boot1.efi -i 2 nda1 will do, but it's not correct command in this case.
Anyway, the handbook can be more clear.
 
I removed the 2 bootia32.efi files and run loaders-update shoot-me -f. All looks fine, only the gpart bootcode has different options for nda1p2 and nda2p2.
Code:
root@AMD_STATION:[~] # loaders-update shoot-me -f
loaders-update v1.2.2

One or more efi partition(s) have been found.

Examining nda2p1...
mount -t msdosfs /dev/nda2p1 /mnt
EFI loader /mnt/EFI/BOOT/BOOTX64.EFI is up-to-date.
EFI loader /mnt/EFI/freebsd/loader.efi is up-to-date.
umount /mnt
Examining nda1p1...
Efi partition nda1p1 is already mounted in /boot/efi.
EFI loader /boot/efi/efi/freebsd/loader.efi is up-to-date.
EFI loader /boot/efi/efi/boot/bootx64.efi is up-to-date.
Examining nda0p1...
mount -t msdosfs /dev/nda0p1 /mnt
There is no FreeBSD loader in nda0p1
umount /mnt

One or more freebsd-boot partition(s) have been found.
The content of the freebsd-boot partition(s) won't be analyzed.
The root file system is zfs.

Examining nda2...
The pmbr on this disk is up-to-date.
About to execute: gpart bootcode -p /boot/gptzfsboot -i 2 nda2
Are you sure (y/N)? y
partcode written to nda2p2
Examining nda1...
About to execute: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda1
Are you sure (y/N)? y
partcode written to nda1p2
bootcode written to nda1

-------------------------------
Your current boot method is UEFI.
Boot device: /dev/nda1p1 File(\EFI\BOOT\BOOTX64.EFI)
Updated BIOS loader: 2
-------------------------------
 
That's ok, I think. Next times you run loaders-update, don't use anymore the -f option as its duty is done now.

I advise to run loaders-update after each freebsd-update upgrade (change of FreeBSD version). Conversely, it's useless after freebsd-update fetch install (simple update of the current version) as the loaders are likely to remain unchanged.
 
I think in case of bootloader problems, everybody will pull the fixed version from a release installer image and not even consider installing a port to upgrade it. But that's probably not a solution because reasons. You can catch a zero-day with full privilege escalation within the bootloader.
As I said, currently, everyone is free to use this or that, or to not care at all about this concern.
The main issue is the zpool upgrade, because once done, if you don't update the loader, you won't boot anymore.

If you are on UFS-on-root, you can live for long time before something bite you from this side.

My point was and always was: the upgrade process should take care of this update. After all, it's an upgrade, so, upgrade all. For example, pfSense does the thing at each upgrade, and that works well.

Unfortunately, pfSense is maybe the only FreeBSD derivative that does that. OPNsense, GhostBSD and others don't. Some users of FreeBSD and its derivatives use loaders-update and are happy such a software exists. I even saw a guy that branched its OPNsense temporaly on the ports repo of FreeBSD to get my software (which is really over-complicated if you compare to a direct fetch to the git repo).
 
My point was and always was: the upgrade process should take care of this update.
100% agreed with this.
It would be nice if:
  1. ZFS pool has feature (or fixed position) to record possible loaders / boot codes outside any ZFS pools that can kick the pool to boot, with supported features by the boot codes / loaders, depending on the installed OS.
  2. zpool upgrade first confirms the recorded possible loaders / boot codes supports read-incompatible features that would be enabled on upgrade or not.
  3. If any of the to-be-enabled read-incompatible feature is NOT supported, confirm updates in /boot/. If not yet, and the pool is bootable one, reject upgrade and error out.
  4. If all updated bootcodes / loaders corresponding to recorded possible ones support all to-be-enabled read-incompatible feature, upgrade boot codes / loaders first, and only when the upgrade is successful, actually upgrade the specified pool.
Boot codes / loaders to be confirmed would be:
  • \EFI\boot\boot{arch}.efi in ESP where {arch} is x64 for amd64
  • \EFI\freebsd\loader.efi
  • \EFI\freebsd\boot1.efi
  • bootcode at partition (in FreeBSD terms, slice) boot record on MBR scheme
  • contents of freebsd-boot partition on GPT scheme
on ALL installed physical drives.

For the first case, \EFI\boot\boot{arch}.efi can be whichever /boot/boot1.efi or /boot/loader.efi.

Boot codes written in the last 2 cases would be:
  • /boot/zfsboot for MBR (already removed at least on stable/15 and main)
  • /boot/gptzfsboot
Note that partition numbers should differ depending on how the first admin installed.
 
T-Aoki, you are one of those I call "knowledgeable people", but kind and open-minded. That being said, it's not always easy to follow you in the details. It appears to me that you added more complexity where I saw some relative simplicity (ok, my solution doesn't work on non-GPT disks).

For what I understand, you propose a means to avoid any starting problem at the base. It's a longer way than just an automatic update. zpool upgrade has been improved in the way that when you upgrade a pool, it checks if bootfs property is defined, and if yes, it warns about the loader specific problem. Better than nothing.

Just let me explain how loaders-update works:

First, its goal isn't to add a loader where previously there wasn't. It updates, never creates (there is an exception for BIOS update in certain circumstances).

The consequence is that you can have an efi loader of whatever name, it finds it. It checks: is this file has an .efi extention? Then, is file reporting this is a PE32+ executable? Is the string "FreeBSD" inside this file? If all the answers are yes, it compares this file with /boot/loader.efi. If there are not the same, it proposes to update it. Consequence of the consequence, it updates all matching files in the ESP partition, even those that are not used to start the machine.

When it finds a freebsd-boot partition, it checks if there is trace of a BTX loader inside. If yes, it try to guess if it is filled by gptboot or gptzfsboot (looking for strings "zfs" and "ZFS"). Once the fs is guessed, it compares with the root file system and only if this is coherent (same fs), it checks if its first boot sector code and the content of the freebsd-boot partition are the same than those in /boot (pmbr and gptboot or gptzfsboot). It proposes the update only for the files that don't match.

Alan Jude reportdly said about loaders-update it's "a bit shot-gun-y". And this is exact. It explores all disks, all the partitions (or almost) and proposes to update each target that falls in the previously evoked conditions. This is to match the case of mirror disks and also because it cannot find in all cases what is the starting disk. But, even in shoot-me mode (unless you used the -y option, beware of that*), you are free to answer yes or no. The admin remains the one that decides. It's why my solution is as simple as safe.


(*) It's perfectly usable when you know for sure that loaders-update found the right targets on your system.
 
I know your sysutils/loaders-update should be handy for many admins having disks in GPT scheme. So I think your choice is sane and reasonable for casual users.

But in environments that many long-standing computers (i.e., replacing parts and keep contents of disks updated but not partitioning scheme for decades) through newly installed computers in GPT-scheme with UEFI boots are mixed up, things are quite complexed and headaches for admins, especially the admin is NOT the exact person installed the oldest one through latest one.

Yes, there can be a plenty of combinations...

This is why I described such a complexed way (would be still too insufficient, though) to make risks for footshooting as few as I could imagine for now.
But to cover "all theoretically possible" situations is too complexed, but it would be what's wanted if let zpool upgrade to update loaders/boot codes. Maybe some admins would want interactive mode, regardless CUI/TUI or GUI as of the complexities.
 
I don't see how a user who has read the handbook can make a "mistake" running the commands specific to their system, maybe a severe case of Emacs pinky or a RS232 line that's a bit too long...
Just to answer to that. You didn't see? Look into this forum and you will find many disasters.
Most of them came from the text we got after zpool upgrade (fortunately, this text has been removed) where it advised to gpart bootcode da0p1 or something like that (I mean BIOS loader update).

Some people were on EFI (including hybrid EFI/BIOS) and killed the msdos file system inside. There are other examples even after they remove this fatal text and yes, it was because these people didn't understand the thing. Newcomers or users that never upgraded the loaders and were obliged to proceed this time for whatever reason. As I already said, the documentation wasn't the same not so long ago. Now, it's far better.

Clearly, I can't say if only one guy left FreeBSD for that reason. I think, it's probable. There are so many OSes that don't bother you with all that stuff and works, just works.

It's for that I wrote loaders-update, not for me. I knew how to do manually. As a side effect, update the loaders is boring and it's more than one time a year. Personally, I have another thing to do than type the commands and triple check them to avoid a... "regrettable error". So, I leverage of my own script even if it wasn't the initial goal.
 
I really don't understand people having a problem with this. A person saw what they felt was a problem, came up with a solution and offers it to the community.
Noone is being forced to use the tool, feedback has been welcomed, discussions have been had.
And what happens? The person providing the tool gets kind of peed on. You don't like the tool, the idea behind the tool? Ok, don't use it. But if someone else finds it useful don't look down on them.

We're talking about a tool that people say "well if you just follow the handbook you don't need the tool" but then we get to "we can't force people to actually read the handbook". It seems like people would rather ask chatgpt or gemini instead of actually reading for themselves.

Emrion I personally think the tool is useful and thank you for the effort to provide it. It may help newbies avoid bad situations, it may save graybeards a few keyclicks. Should it get rolled into something in base? To me that is a different discussion and there are likely pros and cons to be had.
 
Emrion I personally think the tool is useful and thank you for the effort to provide it. It may help newbies avoid bad situations, it may save graybeards a few keyclicks. Should it get rolled into something in base? To me that is a different discussion and there are likely pros and cons to be had.
I spoke of this in the other thread. Considering that a top valuable tool like sysutils/beadm
of vermaden hasn't been integrated in the base, but the FreeBSD staff used a lot of time to get something equivalent in C, you can easily imagine that my poor script will never get in the base.

It's fine for me. They have their reasons, even if I find that questionable (the loss of time). The important thing is the idea and the move to this idea. That begun by the documentation, this will end with the automatic update during the upgrade phase (or why not, a definitive way like T-Aoki imagined).
 
  • Like
Reactions: mer
Back
Top