New FreeBSD 15.1 installation: Cannot open /boot/lua/loader.lua

F

fax668

I am stuck on a new FreeBSD 15.1 installation on a new system and none of the hints in this forum or elsewhere have brought me closer to a resolution. When booting the system in EFI mode (CSM legacy mode is disabled), the ESP loader is giving me this:
Code: 
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk5p1:
FreeBSD/amd64 EFI loader, Revision 3.0

    Command line arguments: rootdev=disk5p2:
    Image base: 0xb5409000
    EFI version: 2.70
    EFI Firmware: American Megatrends (rev 5.17)
    Console: efi (0)
    Load Path: \EFI\FREEBSD\LOADER.EFI
    Load Device: PciRoot(0x0)/Pci(0x2,0x2)/Pci(0x0,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)/HD(1,GPT,8F58E360-8B7B-11F1-BD3D-9C6B009FDCB5,0x28,0x82000)
    BootCurrent: 0000
    BootOrder: 0000[*] 0002
     Setting currdev to configured rootdev disk5p2:
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.

The system has one NVMe SSD for the system installation and four SATA HDDs. Only the NVMe drive has a ESP, i.e. "currdev disk5p1:" clearly is the correct ESP. The system drive has the following partitions:
  • p1: ESP, 260 MB
  • p2: UFS /boot, unencrypted
  • p3: Swap
  • p4: ZFS root, GELI encrypted
  • p5: ZFS L2ARC
Since that partitioning layout isn't fully supported by the installer, I created it manually using a FreeBSD 15.1 live image before running the installer.

Clearly the ESP loader from p1 is being loaded and then it fails to find p2 or the Lua loader on p2. The rootdev setting is obviously picked up and I tried all sorts of combinations from disk1p1: to disk5p5: but disk5p2: should be correct AFAICT.

When I mount p2, I can definitely see the loader.lua file, which is a valid text file that contains a Lua script:
Code: 
-r--r--r--  1 root wheel  2.3K Jun 12 00:37 /tmp/boot/lua/loader.lua

I am wondering one thing now however, because the root of the filesystem on p2 would be /lua/loader.lua, not /boot/lua/loader.lua as it says in the error in the boot console. Did I accidentally screw up the directory levels in p2 when I manually partitioned and created the file systems? Any other hints how to debug this or what I could be looking into before I give up and wipe it all and start over from scratch?
 
Your issue is primarily caused by putting /boot on its own partition. That's common on Linux but the FreeBSD boot process is entirely different.

The FreeBSD boot loaders (loader.efi or gptzfsboot(8)) have enough knowledge of ZFS to be able to boot from it.
 
Alright, I have been trying to emulate how OpenBSD does full disk encryption with a key file on a USB stick, so that the system can boot automatically without waiting for a password. My understanding is that the FreeBSD loader does not support ZFS native encryption with a key file on a USB stick. So I ended up with this unencrypted UFS /boot partition that should then be able to use GELI and decrypt zroot. I skipped that explanation initially because we are not even getting to the stage where the boat loader from the unencrypted /boot partition is invoked. So am I completely on the wrong track here or should that work in principle?
 
fax668 said:
My understanding is that the FreeBSD loader does not support ZFS native encryption with a key file on a USB stick. So I ended up with this unencrypted UFS /boot partition that should then be able to use GELI and decrypt zroot.
Click to expand...
It's possible to boot the encrypted Root-on-ZFS partition with a keyfile from the unencrypted /boot partition. I've done it many time in test setups in VMs.

The loader is looking for /boot/lua/loader.lua, so the root of the boot partitions file system must be /boot. The keyfile configuration is configure in (unencrypted) /boot/loader.conf. See

geli(8) EXAMPLES.
Code: 
           geli_da0_keyfile0_load="YES"
           geli_da0_keyfile0_type="da0:geli_keyfile0"
           geli_da0_keyfile0_name="/boot/keys/da0.key0"
           geli_da0_keyfile1_load="YES"
           geli_da0_keyfile1_type="da0:geli_keyfile1"
           geli_da0_keyfile1_name="/boot/keys/da0.key1"
           geli_da0_keyfile2_load="YES"
           geli_da0_keyfile2_type="da0:geli_keyfile2"
           geli_da0_keyfile2_name="/boot/keys/da0.key2"

           geli_da1s3a_keyfile0_load="YES"
           geli_da1s3a_keyfile0_type="da1s3a:geli_keyfile0"
           geli_da1s3a_keyfile0_name="/boot/keys/da1s3a.key"

     If there is only one keyfile, the index might be omitted:

           geli_da1s3a_keyfile_load="YES"
           geli_da1s3a_keyfile_type="da1s3a:geli_keyfile"
           geli_da1s3a_keyfile_name="/boot/keys/da1s3a.key"
 
Yep, thanks, I have a similar config in loader.conf but clearly the boot is not even getting there.

(Putting a keyfile to an unencrypted partition on the same drive as the zroot seems a bit pointless however.)
 
Code: 
# mkdir /tmp/efi
# mount -t msdosfs /dev/gpt/efiboot0 /tmp/efi
# cat /tmp/efi/efi/freebsd/loader.env
rootdev=disk5p2:
 
fax668 said:
(Putting a keyfile to an unencrypted partition on the same drive as the zroot seems a bit pointless however.)
Click to expand...
The keyfile would be on a removable USB stick (e.g. da0, file system partition labeled "usbkeyf"), the loader would read it from the un-encrypted /boot/loader.conf. Example:
Code: 
geli_usb_keyfile_load="YES"
geli_usb_keyfile_type="nda0p4:geli_keyfile"
geli_usb_keyfile_name="/keys/keyfile.key"

fax668 said:
The system has one NVMe SSD for the system installation and four SATA HDDs. Only the NVMe drive has a ESP, i.e. "currdev disk5p1:" clearly is the correct ESP.
Click to expand...
The number of disks don't add up. "one NVMe SSD" + "four SATA HDDs" makes 5 disks, "currdev disk5p1:" disk numbering starting at "0" makes disk5 a sixth disk.

Please clarify the discrepancy. Perhaps you meant the fifth disk ( disk4p1 )?
 
That's a smart question. I realized that the FreeBSD EFI boot loader console has a lsdev command. Unfortunately, it doesn't really provide any new information:
Code: 
disk devices:
    disk0:    60620800 X 512 blocks (removable)
      disk0p1: FreeBSD UFS
    disk1:    3907029168 X 512 blocks
    disk2:    3907029168 X 512 blocks
    disk3:    3907029168 X 512 blocks
    disk4:    3907029168 X 512 blocks
    disk5:    1000215126 X 512 blocks
      disk5p1: EFI
      disk5p2: FreeBSD UFS
      disk5p3: FreeBSD swap
      disk5p4: FreeBSD ZFS
      disk5p5: FreeBSD ZFS
http: (unknown)
net devices:

disk0 must be the USB stick with the keyfile. disk1 - disk4 are the HDDs. disk5 obviously is the NVMe SSD with the system installation.
 
When I try ls in the EFI boot loader console, I am getting this (honestly don't know how to select the partition on which to run these commands):
Code: 
OK ls /boot
    /boot
OK ls /boot/lua
open '/boot/lua' failed: no such file or directory
OK ls /lua
/lua
    cli.lua
    ....
    loader.lua
    ....
OK

Looks to me like I have the wrong directory structure on the boot partition?
 
Hi fax668,

I didn't go into analyzing your systems specific issue, I did set up a working VirtualBox test VM, simulating your setup. The only difference here is the un-encrypted /boot is on the simulated USB, here a SATA disk, ada0, not on nda0. vbox can provide a virtual USB device, but can't boot from it, that's why using a SATA drive.

You should spot from my test systems settings what causes the issue on your system. I listed only the important parts of the configuration, not including ESP, zpool, etc. creation, system installation, configuration, etc.

Code: 
root@test:~ # sysctl kern.disks
kern.disks: ada4 ada3 ada2 ada1 ada0 nda0

root@test:~ # gpart show -p ada0 nda0
=>      40  33554352    ada0  GPT  (16G)
        40      2008          - free -  (1004K)
      2048  33550336  ada0p1  freebsd-ufs  (16G)
  33552384      2008          - free -  (1004K)

=>       40  524287920    nda0  GPT  (250G)
         40     532480  nda0p1  efi  (260M)
     532520       2008          - free -  (1004K)
     534528    4194304  nda0p2  freebsd-swap  (2.0G)
    4728832  419430400  nda0p3  freebsd-zfs  (200G)
  424159232  100126720  nda0p4  freebsd-zfs  (48G)
  524285952       2008          - free -  (1004K)

Initialize the geli(8) provider for Root-on-ZFS, here nda0p3 ( /keyfiles/keyfile.key is on ada0p1 ).

Make sure to set the -b flag.
Code: 
geli  init  -b  -l 256  -s 4096  -P  -K /mnt/keyfiles/keyfile.key  nda0p3

Simulated, removable USB disk, ada0p1, /boot/loader.conf. For /boot I did cp -a /boot to mount point of mounted ada0p1):
Code: 
geom_eli_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"

geli_nda0p3_keyfile_load="YES"
geli_nda0p3_keyfile_type="nda0p3:geli_keyfile"
geli_nda0p3_keyfile_name="/keyfiles/keyfile.key"

vfs.root.mountfrom="zfs:zroot/ROOT/default"
Without vfs.root.mountfrom the system will stop at a "mountroot>" prompt. This variable has a disadvantage, it won't boot a different ZFS bectl(8) (or beadm) boot environment from the encrypted Root-on-ZFS. If the BE changes, this variable has to be modified accordigly.

ESP efi/freebsd/loader.env
Code: 
rootdev=disk0p1:
Translates to ada0p1.

Properly configured to attach the provider by keyfile, you will see in dmesg(8) the creation of the geli(8) device:
Code: 
GEOM_ELI: Device nda0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: accelerated software
 
T-Daemon said:
on the simulated USB, here a SATA disk, ada0, not on nda0. vbox can provide a virtual USB device, but can't boot from it, that's why using a SATA drive.
Click to expand...
I was mistaken, booting from a VirtualBox virtual USB device works just fine.

In a second test VM I put the "efi" and "boot" partition, including the keyfile, on the da0 virtual USB disk:
Code: 
 # gpart show -p da0
=>      40  67108784    da0  GPT  (32G)
        40    532480  da0p1  efi  (260M)
    532520      2008         - free -  (1004K)
    534528  10485760  da0p2  freebsd-ufs  (5.0G)
  11020288  56088536         - free -  (27G)

efi/freebsd/loader.env: rootdev=disk0p2:
 
Thank you for your work. I first need to figure out why my system is not booting from the unencrypted /boot. The GELI part comes after that. I spent so much time on this now with no end in sight, I think I better start over from scratch and we'll see if I get further. I don't have access to the system this week-end however, will have to wait a few days.
 
fax668 said:
I first need to figure out why my system is not booting from the unencrypted /boot.
Click to expand...
I suspect a missing /boot directory in the un-encrypted "boot" partition. The loader is looking for system boot files under /boot/.
fax668 said:
I am wondering one thing now however, because the root of the filesystem on p2 would be /lua/loader.lua, not /boot/lua/loader.lua as it says in the error in the boot console. Did I accidentally screw up the directory levels in p2 when I manually partitioned and created the file systems?
Click to expand...

Next issue, make sure to put the /boot directory on a partition, sharing the geli(8) keyfile(s) (this excludes the "boot" partition on the "nda" (NVMe) drive).

The loader can't read keyfiles situated in another location than the system /boot directory including partition.

Ideal is to put the /boot directory and keyfiles on the same removable USB device partition.

It would be also possible to bring the /boot directory and keyfiles over PXE, provided the target machines network interface supports wake-on-lan.
 
You must log in or register to reply here.
Back
Top