How to change to alt root filesystem with nanobsd and UEFI?

Bobi B.

Bobi B.

Greetings! I've built several nanobsd systems, performed multiple upgrades (using two root file systems) and, so far, everything worked great. Enter UEFI. With UEFI boot process is handled by /boot/boot1.efi (uefi(8)) and gptboot(8) is no-longer in use. Therefore partition attributes, such as bootme and bootonce, are no-longer taken into account. Therefore atomic upgrades no-longer work. In fact they do work, except newly installed root file system cannot be activated, i.e. is not mounted by the kernel upon boot; instead, first root filesystem is always mounted by the kernel.

Can someone suggest a way to tell /boot/boot1.efi which partition/filesystem to use as a root filesystem? Anyone managed to successfully use alt root filesystem with nanobsd? Dare I ask if some equivalent of bootonce flag is even possible?
 
I have very little experience with UEFI booting and no experience with nanobsd but it's explained in uefi(8):
Code: 
           2.   boot1.efi reads boot configuration from /boot.config or
                /boot/config.  Unlike other first-stage boot loaders,
                boot1.efi passes the configuration to the next stage boot
                loader and does not itself act on the contents of the file.
Then in boot.config(5):
Code: 
     The command:

           # echo "1:ad(1,a)/boot/loader" > /boot.config

     will instruct the second stage of boot(8) on the first disk to boot with
     the third boot(8) stage from the second disk.
After that it just follows the regular loader(8) and loader.conf(5).
 
Unfortunately that doesn't seem to work like the legacy, BIOS way of booting. With MBR boot(8) will look for an active partition. With GPT gptboot(8), will look for a bootonce or bootme partition. With UEFI /boot/boot1.efi will pass configuration to the next stage bootloader, /boot/loader.efi, that will load the kernel from the first filesystem with /boot/defaults/loader.conf.

So, for example, if there are multiple FreeBSD installations on a single drive (say different versions or revisions), one cannot select which one to boot. Any other ideas regarding how to do that with UEFI boot?
 
Ciao, does anyone have any updates?
I saw some kind of support on the
Code: 
../nanobsd/embedded/common
but I'm not sure I understand how it works.
 
Something like this in the update script...

#
# Now update the EFI loader directory to point at the new system point
#

mkdir -p /var/tmp/.new_EFI
mount -t msdosfs /dev/mmcsd0s1 /var/tmp/.new_EFI
ROOT=disk0${PART}
echo "rootdev=${ROOT}" > /var/tmp/.new_EFI/EFI/FreeBSD/loader.env
umount /var/tmp/.new_EFI
rmdir /var/tmp/.new_EFI

echo "Updated to boot $DEST; reboot to activate."

I have a Crochet build that runs on Pi3s which are EFI; same thing should work here. The loader.env file can tell the EFI loader where root is.

(Change devices to suit obviously.)
 
Booting EFI also involves a small bit in BIOS. Like SirDice mentions. efibootmgr is going to need to be invoked to tell BIOS to boot off other EFI device (or partition in this case).
On top of all this efibooting acts different depending on motherboards EFI version.

I have been messing with my "gmirror on EFI" setup and it works different board to board it seems. It takes manual intervention and efibootmgr. Scriptifying it would be tough.

So basically every EFI Boot entry you see in Boot Menu in BIOS is a pointer to an boot file. I will call this entry a UUID/GUID. It points to a specific disk and partition for the EFI file.

Perhaps create a GUID for both entries in NanoBSD and send them to Bios via efibootmgr. Unfortunately there is alot more involved as there are boot priorities.
 
Then how about this. Build into your NanoBSD "flipflop/pingpong" script the EFI Boot Manager setting for BootNext. Then when you reboot it will boot your other partitions EFI file.

efibootmgr

man.freebsd.org man.freebsd.org
-n bootnext variable
The problem is it uses bootnum and not UUID/GUID. So bootnum really depends on the machine BIOS and would be hard to script.
 
The biggest problem I run into with, for example, a firewall appliance is figuring out where it booted from so it can be correctly updated.

I've changed nanobsd's build process (which in legacy.sh was wildly archaic) to produce a dual-bootable (CSM + EFI) media and keep the data partition -- it gets formatted as MBR (thus boots on old hardware that won't boot a GPT USB or SD card, for example) but still has two code partition (s1a and s2a), the EFI partition (s3) and then s4 is a container partition that has both "a" (for the cfg area) and "d" partitions (for data.)

It works nicely however if the device decides that there is another "first" disk (e.g. there's a nVME or mSATA device in there) then the EFI loader may consider the USB Key or SD as "disk1" instead of "disk0" -- which means trying to use the update script is asking to get hosed because it will be wrong.

I've yet to figure out how to RELIABLY get the correct disk number "in EFI's view" from which the CURRENT running system booted -- and that's a problem because the loader can read loader.env BUT it has to be right if the file is present or the machine does not boot without manual intervention on the console.

Absent that file it will find the first bootable partition it can and boot that, but if you want to have dual-partitions for updates obviously you must tell it which one to use.
 
Tickerguy said:
The biggest problem I run into with, for example, a firewall appliance is figuring out where it booted from so it can be correctly updated.
Click to expand...
After the kernel is loaded can't you just check where the root filesystem is mounted from? It's in the output of mount(8).

Tickerguy said:
I've changed nanobsd's build process (which in legacy.sh was wildly archaic) to produce a dual-bootable (CSM + EFI) media and keep the data partition -- it gets formatted as MBR (thus boots on old hardware that won't boot a GPT USB or SD card, for example) but still has two code partition (s1a and s2a), the EFI partition (s3) and then s4 is a container partition that has both "a" (for the cfg area) and "d" partitions (for data.)
Click to expand...
You can perform Legacy BIOS boot (what you call CSM) from a medium with a GPT partition table on it. The reason gptboot(8) and gptzfsboot(8) exist is to support precisely that.

Tickerguy said:
It works nicely however if the device decides that there is another "first" disk (e.g. there's a nVME or mSATA device in there) then the EFI loader may consider the USB Key or SD as "disk1" instead of "disk0" -- which means trying to use the update script is asking to get hosed because it will be wrong.
Click to expand...
The EFI loader tends to know which disk it was loaded from. However, if the update script is not a loader script but a program that runs after the kernel assumes control, it shouldn't matter what disk names the loader uses because the kernel uses different names. The kernel also supports partition labels of various kinds. The loader can tell the kernel to mount its root filesystem from /dev/gpt/first or /dev/gpt/second, for example, without regard for the number of disks in the system or what number each disk got assigned.

Tickerguy said:
I've yet to figure out how to RELIABLY get the correct disk number "in EFI's view" from which the CURRENT running system booted -- and that's a problem because the loader can read loader.env BUT it has to be right if the file is present or the machine does not boot without manual intervention on the console.

Absent that file it will find the first bootable partition it can and boot that, but if you want to have dual-partitions for updates obviously you must tell it which one to use.
Click to expand...
I don't understand what the problem is. If the system booted successfully, why do you care what the disk number "in EFI's view" is?
 
JordanG said:
After the kernel is loaded can't you just check where the root filesystem is mounted from? It's in the output of mount(8).
Click to expand...
That tells me where the system thinks it is but not where the EFI loader thinks it is. And they're not necessarily the same.

Consider a box with a nVME non-bootable device in it you wish to boot from an SD card or USB stick. This under NanoBSD makes for a power-fail-safe device (no risk of the filesystem being inconsistent if the cord is yanked.) For a firewall appliance this is desireable.

Now I want to update the code partition. I have the running one which I can't stomp on, and I know what that is from FreeBSD's perspective so writing the update code to the other one is easy.

But having the loader point at the right one is not. That's the problem, since, for example, the running system here is on /dev/da0s1a however the EFI loader booted it from disk1, not disk0 because disk0 is not bootable.

What I have right now will boot on either a legacy box with no EFI at all (since it has boot0/bootcode on it) or an EFI box. Plug it in, both like it. This is good. And on the legacy box its easy since you just set the active flag on the right partition. The Nanobsd changes made quite some time ago got this sort of but lost the data partition which I resolved by putting it in a BSD partition for the 4th MBR slice.

But on the EFI box its not that simple since the loader, without being told otherwise, will boot the first bootable thing it sees.

covacat said:
you can examine loader's currdev and loaddev with kenv
Click to expand...

Not so sure about that:
Code: 
$ kenv|grep disk
currdev="disk0s1a:"
loaddev="disk0s1a:"
Except its not from the loader's point of view at boot time before it loads the kernel.

I stuck "disk0s2a" in loader.env (which is correct if the loader thinks the USB key is "disk0") and the EFI loader chucked on it when a reboot was attempted. Listing the devices from the loader showed the nVME drive as disk0 which has no bootable partition on it. Manually setting currdev was successful but apparently what the kernel thinks it has now and what the loader on the EFI partition thinks the disk number is differs, and in order to update the loader.env file in the script I need to know what it wants and be able account for the order in which it the EFI loader will enumerate the devices.

Been scratching my head a bit on this one as I figured I could find it in the kernel environment and it would be the same, but it isn't.

This is what the system says once running with gpart:
Code: 
$ gpart show
=>       40  250069600  nda0  GPT  (119G)
         40   67108864     1  freebsd  (32G)
   67108904  182960736        - free -  (87G)

=>       40  250069600  diskid/DISK-BKHD-G20250410045  GPT  (119G)
         40   67108864                              1  freebsd  (32G)
   67108904  182960736                                 - free -  (87G)

=>      63  60125121  da0  MBR  (29G)
        63  11257500    1  freebsd  [active]  (5.4G)
  11257563  11257500    2  freebsd  (5.4G)
  22515063     81920    3  efi  (40M)
  22596983    840517    4  freebsd  (410M)
  23437500  36687684       - free -  (17G)

=>       0  11257500  da0s1  BSD  (5.4G)
         0        16         - free -  (8.0K)
        16  11257484      1  freebsd-ufs  (5.4G)

=>       0  11257500  da0s2  BSD  (5.4G)
         0        16         - free -  (8.0K)
        16  11257484      1  freebsd-ufs  (5.4G)

=>     0  840517  da0s4  BSD  (410M)
       0   62500      1  freebsd-ufs  (31M)
   62500  750000      4  freebsd-ufs  (366M)
  812500   28017         - free -  (14M)
 
Last edited by a moderator:
I start to see the problem. You want to tell the loader which of the two "code" partitions to use on the next boot but the loader's way of identifying disks and partitions isn't helping. The primary problem here, in my opinion, is that the loader doesn't have the support for labels that the kernel has.

Please note that if you know the loader's name for the disk from which the system has booted, you could put that name in loader.env but there isn't a guarantee that on the next boot the disks will be numbered the same.

I suggest trying these approaches to deal with the problem:
  1. If the loader chooses the first partition that looks adequate as the root filesystem, you can use that to your advantage. If entry #3 in the partition table describes Partition A and entry #4 describes Partition B, you can swap them so that entry #3 describes Partition B and entry #4 describes Partition A.
  2. Just like the previous approach but hide (and unhide) one partition from the loader by changing its type. The loader will skip the hidden Partition A and load everything from Partition B, when desired.
  3. Use ZFS. With ZFS the loader's name for a filesystem doesn't contain any disk names or numbers.
  4. Modify the loader to support a special "diskSAME" disk name that refers to the same disk the loader was loaded from. Then tell the loader to use partition "diskSAMEs1a" or "diskSAMEs2a" as desired. This modification to the loader or any other is enabled by the fact that FreeBSD and its loader are open source.
  5. Boot on UEFI systems using the native UEFI way. UEFI has those BootXXX variables stored in non-volatile memory that you can set with efibootmgr(8). The "-k" option allows encoding partition information in the BootXXXX variables which the loader then uses for finding "currdev". The special powers of the "-k" option are not obvious from the man page. This approach is what I actually recommend you try first.
Finally, you should definitely consider using GPT partitioning. It is undeniably simpler to deal with one partition table instead of four. The gptboot(8) boot block has "bootme" and "bootonce" flags that you will like.
 
The problem is that I don't know the name of the disk the loader used and I can't deterministically obtain it once the machine boots. Specifically I have observed it is not the same as any of the visible probes either in efibootmgr nor anywhere in the exposed places via sysctl on a reliable basis. If, for example, there is an nVME drive (without a bootable partition on it or even an EFI partition) it is likely to be found first even though the machine then finds the EFI partition on the USB key and boots loader from it.

If I leave things at defaults it will find the first loadable partition and boot it. That's fine, unless you don't want the first one, then it isn't, and rootdev= lets you specify but you must specify the disk -- and once the machine is running you don't have a deterministic way to know what it is, never mind that someone might remove the nVME drive (or add one) between boots and you don't want that to screw the boot up.

What would solve the problem is a "device-less" syntax in loader.env for "rootdev="; for example, instead of "disk0s1a" if you could specify just "s1a" and have it default to where the loader came from that would work.

But as far as I can tell there is no way to do that; you must specify the entire path. This implies that adding that capacity to the loader would fix it.
 
Tickerguy said:
The problem is that I don't know the name of the disk the loader used and I can't deterministically obtain it once the machine boots. Specifically I have observed it is not the same as any of the visible probes either in efibootmgr nor anywhere in the exposed places via sysctl on a reliable basis. If, for example, there is an nVME drive (without a bootable partition on it or even an EFI partition) it is likely to be found first even though the machine then finds the EFI partition on the USB key and boots loader from it.

If I leave things at defaults it will find the first loadable partition and boot it. That's fine, unless you don't want the first one, then it isn't, and rootdev= lets you specify but you must specify the disk -- and once the machine is running you don't have a deterministic way to know what it is, never mind that someone might remove the nVME drive (or add one) between boots and you don't want that to screw the boot up.
Click to expand...

Knowing what name the loader used for the disk it booted from doesn't help you on the next boot when the order of the disks may be different or there may be an extra disk present or maybe a disk is absent. In other words, knowing the correct disk name from the previous boot doesn't allow you to reliably predict what the correct name for the next boot is. Stop trying to solve the wrong problem.

Tickerguy said:
What would solve the problem is a "device-less" syntax in loader.env for "rootdev="; for example, instead of "disk0s1a" if you could specify just "s1a" and have it default to where the loader came from that would work.

But as far as I can tell there is no way to do that; you must specify the entire path. This implies that adding that capacity to the loader would fix it.
Click to expand...

I presented this idea as approach #4. I offered you four more approaches. Which ones did you read and understand?

I want to add one more approach to the list:
6. Use gptboot.efi(8). It is really similar to gptboot(8) so it will help you unify the UEFI and non-UEFI cases.
 
#4 is what I proposed on the mailing list, and is pretty much what I believe has to be implemented in order to get a deterministic result, because I cannot control what the BIOS will do from one boot to the next. The closest I can come is to note in the release notes for the load that you must set boot order in the BIOS, but having done so that would work both for CSM and EFI boot modes.

I can't use GPT because there are non-EFI machines that won't boot a GPT USB stick at all. With MBR/CSM its easy because the partition set "active" is booted, but in the case of the EFI loader that's ignored and thus it will try to boot the first thing it believes is, unless you tell it otherwise in the loader configuration file.
 
What are those machines that can tell apart GPT's protective MBR from a 1980s MBR? If they don't understand GPT how do they know not to boot when the protective MBR is followed by a GPT partition table?

Also, have you considered approach #1 and approach #2?
 
JordanG said:
Boot on UEFI systems using the native UEFI way. UEFI has those BootXXX variables stored in non-volatile memory that you can set with efibootmgr(8). The "-k" option allows encoding partition information in the BootXXXX variables which the loader then uses for finding "currdev". The special powers of the "-k" option are not obvious from the man page. This approach is what I actually recommend you try first.
Click to expand...
I liked the sound of this one best.
Send two boot variables to EFI via efibootmgr. Ping and Pong. Nanodisk1 and Nanodisk2 or something you could search for with script so regardless of bootnum you have correct partition/disk.
 
Wouldn't you consider NanoBSD to be in maintaince mode? I have seen no improvements since the /custom directory.

Poudriere Image really has taken my favor. With some easy script hacks I have u-boot with correct offsets and stuff I could have never done with NanoBSD.
Plus a unified build environment for embedded or desktops. Long live PKH's work but I think the new tools are better.
 
You must log in or register to reply here.
Back
Top