UEFI system fails to boot loader.efi, but boot1.efi (as bootx64.efi) works fine

J

jem

I have a system that fails to boot using loader.efi(8) but works OK with boot1.efi(8) when installed as BOOTX64.EFI.

My UEFI boot variables are set as follows:
Code: 
Boot to FW : false
BootCurrent: 0001
Timeout    : 1 seconds
BootOrder  : 0001, 0000
+Boot0001* UEFI OS HD(1,GPT,016b678b-d7bf-11ee-bd5e-b42e99e0664b,0x800,0x40000)/File(\EFI\BOOT\BOOTX64.EFI)
                      nda0p1:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
 Boot0000  FreeBSD HD(1,GPT,016b678b-d7bf-11ee-bd5e-b42e99e0664b,0x800,0x40000)/File(\EFI\FREEBSD\LOADER.EFI)
                      nda0p1:/EFI/FREEBSD/LOADER.EFI /boot/efi//EFI/FREEBSD/LOADER.EFI

Unreferenced Variables:

When I installed the system, I copied boot1.efi to /EFI/BOOT/BOOTX64.EFI on my EFI System Partition and the system booted fine using x64 default boot entry (UEFI OS).

As boot1.efi(8) is deprecated in favour of loader.efi(8), I decided to copy the latter to /EFI/FreeBSD/loader.efi and create another boot entry pointing to it. As can be seen above, the device identifier in the Boot0000 entry is identical to the working Boot0001 one, and the path to the EFI binary is correct.

When I set the "FreeBSD" entry to default in UEFI setup and try to boot the system, the display stays off indefinately and judging by the pattern of flashing of the hard disk LED, it's stuck in a reboot loop. I can still get back into the UEFI setup and change the "UEFI OS" entry back to default, then things work again.

The system is built around a Gigabyte B450M DS3H motherboard. It's in native UEFI mode with no CSM enabled. The system has had no problems booting other OS's that provide a EFI boot loader in an OS-specific path, including VMware ESXi and Fedora.

Has anyone else observed this behaviour before and found a root cause?
 
FWIW, I have the same motherboard, and my setup works fine. It looks like this:
Code: 
root@kg-core2:~ # df -h /boot/efi
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada0p1    780K    387K    393K    50%    /boot/efi

root@kg-core2:~ # ls -l /boot/efi/efi/boot /boot/efi/efi/freebsd
/boot/efi/efi/boot:
total 385
-rwxr-xr-x  1 root  wheel  393216 Apr 16  2018 BOOTx64.efi
-rwxr-xr-x  1 root  wheel      12 Apr 16  2018 startup.nsh

/boot/efi/efi/freebsd:
total 0
-rwxr-xr-x  1 root  wheel  0 Mar  6  2022 loader.efi

root@kg-core2:~ # efibootmgr -v | grep -B 1 -A 1 Boot0001
BootOrder  : 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008, 0009, 000A
+Boot0001* UEFI OS HD(1,GPT,689dead3-5e35-11e9-9582-b42e991fc5a7,0x28,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)
                      ada0p1:/EFI/BOOT/BOOTX64.EFI /boot/efi//EFI/BOOT/BOOTX64.EFI
Hope this helps.
 
How old is that efi partition? Did you get an error when copying loader.efi? I ask because in previous versions the efi partition was written using an image file (loader.efifat). That kind of screwed up the filesystem because the image was smaller than the partition. Copying loader.efi would fail even though it should fit in a 200M partition. The way to fix this is to newfs_msdos(8) the efi partition and recreate the necessary directories.
 
tingo , your post indicates that your system is set to boot from efi/boot/BOOTx64.efi, not from efi/freebsd/loader.efi. Also, your loader.efi file is showing as 0 bytes long. What version of FreeBSD are you using, because on 14.0-RELEASE, neither loader.efi or boot1.efi are 393216 bytes long.

SirDice , my ESP is 128MiB in size. I manually created the filesystem on it with newfs_msdos(8), mounted it, then copied the .efi files into it.
Code: 
[root@hyp0 ~]# fstyp /dev/nda0p1
msdosfs

[root@hyp0 ~]# df -h /boot/efi
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/nda0p1    128M    832K    127M     1%    /boot/efi

[root@hyp0 ~]# find /boot/efi -ls
     1        0 drwxr-xr-x    1 root    wheel     16384 Jan  1  1980 /boot/efi
  2576       16 drwxr-xr-x    1 root    wheel      8192 Mar  1 11:39 /boot/efi/EFI
  2832       16 drwxr-xr-x    1 root    wheel      8192 Mar  1 11:39 /boot/efi/EFI/BOOT
  2834      320 -rwxr-xr-x    1 root    wheel    157184 Mar  1 11:39 /boot/efi/EFI/BOOT/BOOTX64.EFI
  8208       16 drwxr-xr-x    1 root    wheel      8192 Mar  1 11:40 /boot/efi/EFI/FreeBSD
  8211     1296 -rwxr-xr-x    1 root    wheel    659968 Mar  1 11:40 /boot/efi/EFI/FreeBSD/loader.efi
 
Nothing out of the ordinary, GPT scheme, first partition is an EFI system partition:
Code: 
[root@hyp0 ~]# gpart show nda0
=>       40  488397088  nda0  GPT  (233G)
         40       2008        - free -  (1.0M)
       2048     262144     1  efi  (128M)
     264192  488132936     2  freebsd-zfs  (233G)
 
I meant gpart show without disk specification to see the others disks, if you have, and specifically others ZFS places.
 
The other disks contain whole-device ZFS pools. There's no partition scheme, ESP, or EFI binaries on them.

The full gpart show output only shows some zvols from those pools that are used for virtual machine storage:
Code: 
[root@hyp0 ~]# gpart show
=>       40  488397088  nda0  GPT  (233G)
         40       2008        - free -  (1.0M)
       2048     262144     1  efi  (128M)
     264192  488132936     2  freebsd-zfs  (233G)

=>      34  33554365  zvol/vmstore2/testvm  GPT  (16G)
        34      2014                        - free -  (1.0M)
      2048    262144                     1  efi  (128M)
    264192   2097152                     2  linux-data  (1.0G)
   2361344  31191040                     3  linux-data  (15G)
  33552384      2015                        - free -  (1.0M)

[root@hyp0 ~]# zpool list -v
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
system      232G  1.30G   231G        -         -     0%     0%  1.00x    ONLINE  -
  nda0p2    233G  1.30G   231G        -         -     0%  0.56%      -    ONLINE
vmstore0    111G   423K   111G        -         -     0%     0%  1.00x    ONLINE  -
  ada0      112G   423K   111G        -         -     0%  0.00%      -    ONLINE
vmstore1    444G  2.77G   441G        -         -     0%     0%  1.00x    ONLINE  -
  ada1      447G  2.77G   441G        -         -     0%  0.62%      -    ONLINE
vmstore2    444G  1.99G   442G        -         -     0%     0%  1.00x    ONLINE  -
  ada2      447G  1.99G   442G        -         -     0%  0.44%      -    ONLINE
 
So, you took 3 whole disks for ZFS pools... Well, that could be the culprit. And if it is, it's a bug from loader.efi. It's not advised anymore to take a raw disk for a pool.

If you want to verify this hypothesis, Unplug those tree drives and retry to start on loader.efi.
 
The device that FreeBSD is booting from is not a whole-disk pool. loader.efi is sitting in a simple EFI system partition on a GPT scheme disk.

I've been running whole-disk pools on non-boot drives ever since ZFS was added to FreeBSD. I was pretty certain that's not the problem. Nevertheless, I went into my UEFI setup and disabled the SATA controller so the three adaX drives were not detected. It made no difference. The system still failed to boot using loader.efi.

Other things I just tried:

  • Copied loader.efi to [ESP]/EFI/BOOT/BOOTX64.EFI, overwriting the copy of boot1.efi that was there before. Same result, the system wouldn't boot with the "UEFI OS" entry either.
  • The motherboard's UEFI setup has a "boot override" menu where you can perform a one-time boot from any of the defined EFI boot entries. Strangely, if I select an entry that points at loader.efi from this list, it boots fine.
I'm now pretty sure this is a bug in Gigabyte's UEFI implementation. I would report it to them, but I expect they'll see "FreeBSD" and immediately say "not supported, won't fix.
 
Currently I have not enough time to look into source files, so ths is based on old memories of boot1.efi and prediction about loader.efi.

boot1.efi walks through all attached drives which UEFI firmware recognizes, and for each drives starting from currently booted drive, look for ZFS pool containing /boot/loader.efi, then, UFS partition containing /boot/loader.efi.
Drives are checked currently booted drive first, then all drives UEFI firmware recongizes the order as UEFI firmware recognized.
Lastly, basically first-match is preferred. If I recall correctly, boot1.efi doesn't aware of ZVOLs in each ZFS pools.

I formerly thought loader.efi would be doing the same, but in Sep.2022, a person (not me) did relatively extensive test reported at freebsd-users-jp ML in Japanese and found loader.efi preferres UFS over ZFS, unlike boot1.efi.

Unfortunately, the test didn't include ZVOLs, so if loader.efi is aware of ZVOLs, possibly ESP in the ZVOL on ada0-3 drives causes problem.
Sorry, not certain enough.
 
You must log in or register to reply here.
Back
Top