Utility that tries to figure how to update the FreeBSD loaders

E

Emrion

Hello all,

I worked on this subject these days and finally released this sh script. I would like to know what is your feeling about this idea and possibly your feedback after trying it.

github.com

GitHub - Emrion/uploaders: A proof of concept concerning the automatic update of FreeBSD loaders

A proof of concept concerning the automatic update of FreeBSD loaders - Emrion/uploaders
github.com github.com

Content of README:
This sh script is a proof of concept that an automatic update of FreeBSD loaders is possible.

Update of the loaders is linked to the evolution of zfs features, but not only. Even if no new features are added and the zpool(s) not upgraded, from time to time, some weird problems arise if you don't update. Not to speak about the correction of the loaders bugs, which concerns also the systems with ufs on root. Some people told me that an automatic update of the loaders isn't possible or, at least, not desirable. Too complex is that thing, they said... So, I wrote this script to demonstrate the opposite.

I admit, I put some serious limitations in this code, for I wanted to stay in the fields I know well.

It updates nothing but tells what it would do.

What it does handle:

  • Checks all the disks reported by the system (case of mirror disks).
  • EFI and BIOS loaders.
  • EFI partitions mounted or not mounted.
  • Checks for the presence of EFI loaders both in efi/boot and efi/freebsd.
  • Recognizes (or try) to identify FreeBSD EFI loaders and would update only them.
  • Doesn't change the name of EFI FreeBSD loaders (case where the admin would have changed the default ones).
  • In case of freebsd-boot partition, checks the coherence between its content and the root file system.

What it doesn't handle:

  • Other architecture than amd64.
  • Disks that have other scheme than GPT (concerns mainly MBR scheme).
  • Not enough room in the efi partition to copy the loader (can arise with installed version 12 or before and never updated the loader).
  • More than one efi or freebsd-boot partition a disk. It examines only the first efi and freebsd-boot partition.
Please, try out this sh script on your machines and report back the encountered problems along with your detailed configuration.
Click to expand...

Note that it seems to work also on GhostBSD.
 
If a loader goes bad, installing it to all disks in one step would not be desirable. Similarly there can be efi and mbr compatibility differences. If adding ability to perform the updates, an option to limit updating to 1 new write at a time so it can be individually tested before overwriting others would be nice. I hope 'uploaders' won't be a final name as that may make looking it up get blended into the other search results of 'upload' and its variations with suffixes. If it won't be able to do the job of writing a new boot loader then I'd say the name is misleading at present.

"One or more" statement can be calculated by knowing the #s of the previous two lines but the line is just redundant information with those lines above it saying it but in more detailed count. 'updatable' (I prefer spelling with "tae" instead of "ta", or just rewrite to avoid like "can be updated"...all a moot point if the output gets simplified.

I'd consider simplifying output to not have blank lines, dashed lines (though this would be a good place to title the content in the section).

"Root fs: zfs / Partition has: ufs." ...hooray, you helped me find a part of my setup to review! The part of the script that checks that reads the freebsd-boot section 3 times: once to see if it can be read, once to see if it contains ZFS and once to see if it contains zfs.
Replacing
Code: 
        p="/dev/${d}p$index"
        cat $p > /dev/null
        if [ $? -ne 0 ]; then
            echo "Error during access to $p. Won't update these loaders."
            Err=$(($Err+1))
            continue
        fi
        r1="$(cat $p | grep -c ZFS)"
        r2="$(cat $p | grep -c zfs)"
        if [ $r1 -gt 0 ] && [ $r2 -gt 0 ]; then
            SL="$Sgptzfsboot"
            nfs="zfs"
        fi
        if [ $r1 -eq 0 ] && [ $r2 -eq 0 ]; then
            SL="$Sgptboot"
            nfs="ufs"
        fi
with
Code: 
      r1=`grep -ce ZFS -e zfs  /dev/${d}p$index 2>/dev/null`
      if [ $? -ne 0 ]; then
            echo "Error accessing $p. Loader not checked."
            Err=$(($Err+1))
            continue
      else
        if [ $r1 -gt 0 ]; then
            SL="$Sgptzfsboot"
            nfs="zfs"
        else
            SL="$Sgptboot"
            nfs="ufs"
        fi
...[rest of the function that does things with these checks]
      fi
That uses 1 partition read instead of 3, 1 grep on 1 read instead of 2 on 2 and I started simplifying tests as a result. I prefer to not reach lines of code that aren't useful in a functions pass instead of running across all of them and intentionally throwing away their output; I haven't benchmarked such performance logic for /bin/sh specifically. I don't know if there other advantages/disadvantages to using "$()" instead of "``" when executing a function but `s saves 1 byte per occurence. On a side note, I was surprised how much faster grep with two -e statements ran compared to `grep -c "ZFS\|zfs" /dev/ada0p1` which took many times longer with a maxed out cpu core load.

Wasn't found on my run but in script review, consider rewording "Did find no" as "Found no". I think there were other technicalities like efi vs EFI eventually leading to changing "efi partition" to "ESP", "ESP (EFI system partition)", or just "EFI system partition" (like the loader.efi manpage uses).

"Cannot mount nda0p1 so cannot update its loader(s)." could be changed to make it more clear that the efi loader is what was not able to be checked and that it couldn't be checked instead of it couldn't be updated.
 
First, you must understand that when you boot, you use only one loader. This is a non-sense to leave one partition without update. In case of mirror disks, this can bite you if the main disk fails and you forgot to update the loaders of the others disks. If you really have a problem, you can boot on a stick. That said, I never had any trouble to update the loaders all at once.

"uploaders" is just a name. Nothing final and I didn't think about the possibly impact of that spelling. On my FreeBSD systems, I named it 'u', for it is more quickly typed. This is more a concept than a tool. But, after releasing it, I realized this can help people to know what is the right command line(s) to type.

Thank you for pointing out some optimizations, as I'm far to be an expert in sh scripting. But, do you really think that the speed of execution is of any importance here?

I saw that grep returns 1 when it didn't find the search pattern. But, I can change the test to check a result > 1. I'm wondering, if in case of impossibility to read the partition what it would return in fact. I can test it.

Concerning the English writing, I'll take you comments in consideration, for English isn't my mother tongue and I would like a perfect writing like I can do in french.

I'm very grateful for your glance on this work.
 
I commited several changes including the removal of unnecessary cat.

Personally, I prefer to read a more aerial output, so I left the blank and dots lines.
I didn't understand what you mean by "spelling with 'tae' instead of 'ta'". Updatable is a correct english word, but updataeble isn't. The idea behind the use of "updatable" is to make clear that no change has been made in the system.

The real weakness of this script is the recognition of the loaders. For the efi one, I search the string 'FreeBSD' inside the binary file. I think, it's somewhat robust. But, for the BIOS loaders, I think it's not sufficient. gptzfsboot is recognized just because of the presence of strings 'ZFS' and 'zfs'. And worse for gtpboot, as nothing clearly identifies it. The logic is: if it's not gptzfsboot, it's gptboot. It's insufficient from my point of view.
 
I corrected and simplified the code, add the feature to detect if the loaders are up to date or not.

Now, this utility is more aimed to help people to test if their loaders are up to date and, if not, to give the command lines to type in order to update them.
 
When you suggest copying the new boot.efi file you could suggest cp -p instead of just cp to preserve the file date. Also remove the full stop at the end of the suggested command to make it easier to cut and paste.

This is just personal preference so feel free to ignore but I would prefer the output to be a little less verbose. In particular, there is no need to explain that the script only works on amd64 etc. Just run silently on amd64 and stop with an error if not amd64. Likewise, for root access - it's enough to print an error and explain why root is necessary. For accessing disk devices in /dev, adding to the operator group (pw groupmod -m operator <user>) could be an alternative suggestion.

A larger change that you could consider is if the opening paragraph and prompt to start is necessary. If you are not making any changes, then such a warning makes the user nervous. If a change is needed (like mounting) then show an error and suggest running the command with a new command line option to allow the mount to take place.

The message "But no loader seems to be updatable." implies an error. If all checks have passed, maybe something like "All loaders are up-to-date." would be better.

Finally, I know this is a work-in-progress, but some comments at the top with version, author, license etc would be helpful.
 
Emrion said:
… add the feature to detect if the loaders are up to date or not. …
Click to expand...

Also:

1722419842305.png


The large alert (lines 557-563) appears before the loader menu.

The menu presents a more discreet alert.

I have suggested changes to both alerts.
 
kjeacle said:
When you suggest copying the new boot.efi file you could suggest cp -p instead of just cp to preserve the file date.
Click to expand...
Why? "-p" preserves also the original owner user and group, but anyway you can only copy here with the root user. So cp without option is sufficient. Who cares about the date?

kjeacle said:
it's enough to print an error and explain why root is necessary. For accessing disk devices in /dev, adding to the operator group (pw groupmod -m operator <user>) could be an alternative suggestion.
Click to expand...
Good point. I overlooked that all my users or almost belong to the operator group. With a user that doesn't, the process is stopped when it tries to recognize if a freebsd-boot partition is filled with gptzfsboot or gtpboot, because it can't read it.

I should write this possible cause on the console. I don't know yet. But, if your user is not of the operator group, it's simpler to retry the script with root user than add this user to this group.

For the rest, I will consider your others remarks, in particular the prompt to start, which is indeed of little use.

The author and the licence in the code isn't of a great deal for me. I put a BSD-2-Clause license because git wants one. Maybe, I will change my mind one day.

Thank you very much for examining my work.
 
Cath O'Deray said:
Also:

View attachment 19771

The large alert (lines 557-563) appears before the loader menu.

The menu presents a more discreet alert.

I have suggested changes to both alerts.
Click to expand...
It's a good point as it will alert on the root cause of this trouble: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279829

Nevertheless, it's like a a plaster on a wooden leg. For this is not the only problem you get when you leave your loaders without update. I'm convinced that the update of the loaders must be, at least, proposed (if not automatic) during the process of upgrading.
 
Emrion said:
Why? "-p" preserves also the original owner user and group, but anyway you can only copy here with the root user. So cp without option is sufficient. Who cares about the date?
Click to expand...
Because it provides a simple visual check that the file is from the distribution without having to calculate checksums:
Code: 
% ls -l /boot/efi/efi/freebsd/loader.efi /boot/loader.efi
-r-xr-xr-x  1 root wheel 660480 Jun  6 19:00 /boot/efi/efi/freebsd/loader.efi
-r-xr-xr-x  2 root wheel 660480 Jun  6 19:00 /boot/loader.efi
Regardless of licensing, including author and version information is useful to check where the script came from and if the most recent version of the script is installed. Including the GitHub URL would make sense too. So either printing this information to the console or just adding it as a comment would definitely be beneficial (especially on a multi-user system where the end-user may not be the person who installed it).
 
A new evolution of this script...

Edit:
- The shoot-me mode works on GhostBSD. ReEdit: this OS uses efi/ghostbsd/BOOTX64.efi as loader. So that doesn't work in fact. I have to write a code to explore all the directories inside efi/ (*).
- This very mode works even on pfSense. I explain a little: I tested it on a pfSense VM that was installed with EFI enabled. The EFI loaders were up to date (it's automatic under pfSense), both in efi/boot and efi/freebsd, but that wasn't the case concerning the freebsd-boot partition. So, my script updated this last, after what I make it booted in BIOS mode with success.
- And that works also on TrueNAS-13.0-U6.2. There is an error at some point because /mnt is used (mounted) but without consequence. I use this directory for mounting the EFI partition in case it isn't already mounted in /boot/efi.
 
Last edited:
As a result of (*), a new version of loaders-update arose. It lists all the files inside the efi partition and works on all of them.
 
Hi Emrion , I just tried Emrion/uploaders on 2 personnal servers. One at home and the other at OVH. It seems ok for me.
On booth I've zroot on raidz1 or raidz2 so multiple efi part...
I will try it at work on my personnal workstation, and then on the 3 server on which I've FreeBSD for ZFS NFS export and backup.

Good job.
 
Hi, I just launch it on my station at work.
Code: 
root@station:~# freebsd-version
14.0-RELEASE-p9
Code: 
root@station:~# zpool status
  pool: zroot
 state: ONLINE
  scan: resilvered 45.6M in 00:00:00 with 0 errors on Tue Dec  5 09:46:55 2023
config:

    NAME                STATE     READ WRITE CKSUM
    zroot               ONLINE       0     0     0
      mirror-0          ONLINE       0     0     0
        gpt/zroot-nvme  ONLINE       0     0     0
        gpt/zroot-sata  ONLINE       0     0     0

errors: No known data errors
Code: 
root@station:~# gpart show -p
=>        40  1000215136    nda0  GPT  (477G)
          40      524288  nda0p1  efi  (256M)
      524328     8388608  nda0p2  freebsd-swap  (4.0G)
     8912936   991302232  nda0p3  freebsd-zfs  (473G)
  1000215168           8          - free -  (4.0K)

=>        40  1000215136    ada0  GPT  (477G)
          40      524288  ada0p1  efi  (256M)
      524328     8388608  ada0p2  freebsd-swap  (4.0G)
     8912936   991302232  ada0p3  freebsd-zfs  (473G)
  1000215168           8          - free -  (4.0K)
Code: 
root@station:~# bash ./loaders-update show-me
This utility helps to update the FreeBSD loaders.
show-me mode: it tells what it would do but changes nothing.

One or more efi partition(s) have been found.

mount -t msdosfs /dev/ada0p1 /mnt
EFI loader /mnt/EFI/Boot/bootx64.efi is up to date.
umount /mnt
mount -t msdosfs /dev/nda0p1 /mnt
EFI loader /mnt/EFI/Boot/bootx64.efi is up to date.
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nda0p1 File(EFI\boot\bootx64.efi)
One or more target partition(s) have been found...
All loaders are up to date.
-------------------------------
Code: 
root@station:~# bash ./loaders-update shoot-me
This utility helps to update the FreeBSD loaders.
shoot-me mode selected!

One or more efi partition(s) have been found.

mount -t msdosfs /dev/ada0p1 /mnt
EFI loader /mnt/EFI/Boot/bootx64.efi is up to date.
umount /mnt
mount -t msdosfs /dev/nda0p1 /mnt
EFI loader /mnt/EFI/Boot/bootx64.efi is up to date.
umount /mnt

-------------------------------
Your current boot method is UEFI.
Boot device: nda0p1 File(EFI\boot\bootx64.efi)
One or more target partition(s) have been found...
All loaders are up to date.
-------------------------------
 
You must log in or register to reply here.
Back
Top