ZFS Problems with ZFS root, uefi boot

V

vstemen

Hi.

I am trying to setup a FreeBSD 14.0-RELEASE system, with uefi boot, to mount a zfs file system as root that also has a ufs partition containing an alternate root fs as a fall back. I am using loader.conf to configure it to boot using the zfs root fs. I want to be able to do that without having dependencies on the ufs root if possible. Or in worse case, not rely on the ufs root for anything but loader.conf.

I am having several problems that I hope somebody can shed some light on.
I do have loader.efi installed in the uefi fs as /efi/boot/bootx64.efi.That part is working fine.

My /boot/loader.conf.d/loader.conf file contains

zfs_load=YES​
vfs.root.mountfrom="zfs:zroot/root1"​

First, I can only get loader to read the loader.conf from the ufs file system. Is there any way to get it to read loader.conf from the zfs file system? It seems this would somehow have to be configured in the uefi boot. Ideally, it seems that I should have the ability to put a configuration file for loader in the uefi file system but I have found no documentation indicating that is possible.

This allows it come up with zroot/root1 mounted on root (/). But the problem is, it still loads the kernel and modules from the ufs file system.

If I put
currdev="zfs:zroot/root1:"​

in loader.conf, it panics on boot, does a hex dump of some data, and says,

panic: free: guard1 fail @ 0x6b540f30 from unknown:0​

Yet, without that in the loader.conf, I can drop to the loader prompt at boot, by hitting option 3 in the loader menu, and manually enter

set currdev="zfs:zroot/root1:"​

then run "autoboot", and it boots into the zfs root just fine and also loads the kernel and modules from zfs like I want.

I am having a hard time finding any complete up to date documentation with working examples. What few examples I have found show the colon at the end of the currdev setting. I did find a comment or two in forums where somebody said, "Don't forget the colon at the end", but nobody says why. The

vfs.root.mountfrom="zfs:zroot/root1"​

setting doesn't require it. There doesn't seem to be any colon required at the end of any of the other settings. So why does currdev require it? I confirmed it is required. It didn't work without it.

Although it is not as simple as it should be, I can live with having to reconfigure loader.conf to switch between different root file systems, but I need it to completely switch and not load some stuff from one and the rest from another (which might not even be the same version).

Although not ideal, I can even live with loader.conf always being read from the ufs file system, even though I am booting to zfs, but it is imperative that it load the kernel and modules from the same file system that it is mounting as root.
 
vstemen said:
… I am having a hard time finding any complete up to date documentation with working examples.
Click to expand...

1722998521072.pngNot as hard as when there's the loader alone, with no other computer (to seek documentation).

Pictured: me struggling a few days ago. Of course, I gave up. I would never have guessed a trailing colon.

What few examples I have found show the colon at the end of the currdev setting. …
Click to expand...

See loader_simp(8)
 
*When* you can boot successfully, do you see "/boot/efi/efi/freebsd/loader.efi"? AFAIK it should be the same as "/boot/loader.efi". If you do "df /boot/efi" you should see filesystem "/dev/gpt/efiboot0" or something. If you do "gpart list" one of the disk partition should have "label: efiboot0" and "type: efi". This partition should be quite large (mine is 200MB). "man loader.efi" to learn a bit more about it.
 
vstemen said:
vfs.root.mountfrom="zfs:zroot/root1"
Click to expand...

zpool get bootfs zroot

What's reported?

vstemen said:
… trying to setup a FreeBSD 14.0-RELEASE system, …
Click to expand...

bakul said:
"man loader.efi"
Click to expand...

Side note: the EXAMPLES section is outdated for "recent" installations of the OS; where, for example, installation originated with 14.0-RELEASE or greater.

bakul said:
… should see filesystem "/dev/gpt/efiboot0" or something. …
Click to expand...

I have a "something" variant :-) that does not yet use the label, only because I lazily edited my fstab years after installing a CURRENT that predated 14.0-CURRENT.

Code: 
% df -h /boot/efi
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada1p1    260M    1.3M    259M     1%    /boot/efi
% gpart show -l ada1
=>        40  1953525088  ada1  GPT  (932G)
          40      532480     1  efiboot0  (260M)
      532520        2008        - free -  (1.0M)
      534528    33554432     2  swap0  (16G)
    34088960  1919434752     3  zfs0  (915G)
  1953523712        1416        - free -  (708K)

% grep boot /etc/fstab
/dev/ada1p1             /boot/efi               msdosfs    rw                         0     0
% uname -KU ; uname -a
1500023 1500023
FreeBSD mowa219-gjp4-zbook-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT main-n271593-d8a16b6a4c44 GENERIC-NODEBUG amd64
%

How recent is recent?

base 0b7472b3d8d2f1e90fade5236b44fd98d8e396c2 and base 7919c76dbdd20161247d1bfb647110d87ca5ee0f were on branches main, release/14.0.0 and release/14.1.0. Respective commit dates:
  1. 2021-02-23 – Mount the EFI system partition (ESP) on newly-installed systems.
  2. 2022-05-24 – bsdinstall: use gpt/efiboot0 label in /etc/fstab
HTH
 
vstemen said:
I am trying to setup a FreeBSD 14.0-RELEASE system,
Click to expand...
Be advised, 14.0 will be end-of-life on Sept. 30.



vstemen said:
My /boot/loader.conf.d/loader.conf file contains

vfs.root.mountfrom="zfs:zroot/root1"
Click to expand...
Don't use vfs.root.mountfrom. That variable specifies the root file system to mount, but not necessarily loads the kernel from the same file system.

The FreeBSD loader boots the first kernel found in line. If the system has multiple partitions containing a kernel, the first kernel found will be loaded.

This is problematic if the systems have different kernel versions, patch level, configuration. This is what you are experiencing now:
vstemen said:
This allows it come up with zroot/root1 mounted on root (/). But the problem is, it still loads the kernel and modules from the ufs file system.
Click to expand...
vstemen said:
it is imperative that it load the kernel and modules from the same file system that it is mounting as root.
Click to expand...
Use the currdev variable. This will load the kernel from the same root file system the kernel is installed on.


vstemen said:
I can only get loader to read the loader.conf from the ufs file system. Is there any way to get it to read loader.conf from the zfs file system? It seems this would somehow have to be configured in the uefi boot. Ideally, it seems that I should have the ability to put a configuration file for loader in the uefi file system
Click to expand...
Forget loader.conf, use /boot/efi/efi/freebsd/loader.env.

This is not documented, but when the system is booted, you can see (not under normal circumstances, text scrolls to fast to read) EFI console reading loader environment variables from loader.env:

efi_console_loader.env.png


efi_console.png



vstemen said:
Although it is not as simple as it should be, I can live with having to reconfigure loader.conf to switch between different root file systems,
Click to expand...
You don't need to reconfigure boot environment configuration files manually every time to switch between file systems.

With shell aliases you can create nextboot(8)-like UFS / ZFS boot configurations.

On both systems, make sure /boot/efi is mounted (preferable from fstab), create shell aliases:

Boot environment UFS. Adjust the partition number in disk0p4 to the actual partition number of your system:
Code: 
alias  beu="echo currdev=disk0p4: > /boot/efi/efi/freebsd/loader.env"

Boot environment ZFS:
Code: 
alias  bez="echo currdev=zfs:zroot/root1: > /boot/efi/efi/freebsd/loader.env"

With aliases and loader.env you can boot several UFS / Z file systems from one disk (provided ZFS's have different pool names).
 
T-Daemon said:
Don't use vfs.root.mountfrom. That variable specifies the root file system to mount, but not necessarily loads the kernel from the same file system.

Use the currdev variable. This will load the kernel from the same root file system the kernel is installed on.
Click to expand...
That's what I was trying to use, but, as I mentioned, when I do, loader panics when booting with
panic: free: guard1 fail @ 0x6b540f30 from unknown:0​
T-Daemon said:
Forget loader.conf, use /boot/efi/efi/freebsd/loader.env.

This is not documented, but when the system is booted, you can see (not under normal circumstances, text scrolls to fast to read) EFI console reading loader environment variables from loader.env:
Click to expand...
Thank you! That is what I was looking for, if I can get it to work. That is something that desperately needs documented. It might have saved me a lot of time and work. However, I tried putting currdev="zfs:zroot/root1:" in the efi fs in /efi/freebsd/loader.env but it had no effect. I didn't get the error though. It seemed to not read that file at all. Even if I put vfs.root.mountfrom="zfs:zroot/root1" in it, it still booted to the UFS file system.

Maybe I should upgrade to freebsd 14.1 for further testing. Is it possible that these problems have been fixed since 14.0?

T-Daemon said:
You don't need to reconfigure boot environment configuration files manually every time to switch between file systems.

With shell aliases you can create nextboot(8)-like UFS / ZFS boot configurations.

With aliases and loader.env you can boot several UFS / Z file systems from one disk (provided ZFS's have different pool names).
Click to expand...
Thanks. Yea, I already planned to use scripts or aliases to make it easy to switch once I got it working and confirmed what needed to be done. I just didn't elaborate on it to keep my original posting from getting so long :).
 
bakul said:
man loader.efi
Click to expand...

T-Daemon said:
… This is not documented, but when the system is booted, you can see (not under normal circumstances, text scrolls to fast to read) EFI console reading loader environment variables from loader.env: …
Click to expand...

Documented in the manual:

"… loader.efi loads some extra variables early in startup from /efi/freebsd/loader.env from the EFI partition. …"

(Many things scroll past too fast to read at startup time …)
 
Cath O'Deray said:
Not as hard as when there's the loader alone, with no other computer (to seek documentation).

Pictured: me struggling a few days ago. Of course, I gave up. I would never have guessed a trailing colon.
Click to expand...
Ha ha. Yea, it's almost impossible to do this stuff without another computer for documentation :)

Cath O'Deray said:
See loader_simp(8)
Click to expand...
Yea, I actually discovered this manual on the test machine where I am trying to get this to work while I was writing my original posting here. I was originally using the man pages on my freebsd 13.1 machine and found no mention of the currdev setting, outside of web searches, until I started going over the man pages on the 14.0 test machine.
 
Cath O'Deray said:
Documented in the manual:

"… loader.efi loads some extra variables early in startup from /efi/freebsd/loader.env from the EFI partition. …"

(Many things scroll past too fast to read at startup time …)
Click to expand...
It's not in my loader.efi manual on freebsd-14.0. You must be on a newer release. loader.env is probably not implemented yet then. The would explain why it had no effect putting settings in it. That clinches it. I think I need to go ahead and upgrade before I go any further.
Thanks.
 
Cath O'Deray said:
Compare with mine (a result of bectl activate 1500023-010-base):

Code: 
% zpool get bootfs august
NAME    PROPERTY  VALUE                         SOURCE
august  bootfs    august/ROOT/1500023-010-base  local
%
Click to expand...
Looks like the same result. Thanks. You just taught me about bectl. I didn't know of it's existence. The man page says it was derived from beadm, which I just discovered yesterday when the chatgpt ai mentioned it while I was using chatgpt to try to help resolve the booting issues. I Guess now I'm going to have to study bectl to see how I like it :-).
 
This one is interesting. Here's my test setup under VM. I manage to switch the boot partition with the following

To boot from ZFS partition
set currdev="zfs:zroot/ROOT/default:"

To boot from UFS
set currdev=disk0p3:
Click to expand...

Here's the entire setup if someone want to try it. It's EFI only.

Code: 
zpool destroy "zroot"
sysctl vfs.zfs.min_auto_ashift=12
gpart destroy -F "da0"
graid delete "da0"
zpool labelclear -f "/dev/da0"
gpart create -s gpt "da0"
gpart add -a 4k -l efiboot0 -t efi -s 260M "da0"
newfs_msdos "/dev/gpt/efiboot0"

gpart add -a 1m -l swap0 -t freebsd-swap -s 2G "da0"
gpart add -a 1m -l ufs0 -t freebsd-ufs -s 20G "da0"
gpart add -a 1m -l zfs0 -t freebsd-zfs "da0"

newfs -U -L ufs1 -S 4096 /dev/da0p3

zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f "zroot"   da0p4
zfs create -o mountpoint=none "zroot/ROOT"
zfs create -o mountpoint=/ "zroot/ROOT/default"
zfs create -o mountpoint=/home "zroot/home"
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "zroot/tmp"
zfs create -o mountpoint=/usr -o canmount=off "zroot/usr"
zfs create -o setuid=off "zroot/usr/ports"
zfs create  "zroot/usr/src"
zfs create -o mountpoint=/var -o canmount=off "zroot/var"
zfs create -o exec=off -o setuid=off "zroot/var/audit"
zfs create -o exec=off -o setuid=off "zroot/var/crash"
zfs create -o exec=off -o setuid=off "zroot/var/log"
zfs create -o atime=on "zroot/var/mail"
zfs create -o setuid=off "zroot/var/tmp"
zfs set "mountpoint=/zroot" "zroot"

mkdir -p "/mnt/tmp"
chmod 1777 "/mnt/tmp"
mkdir -p "/mnt/var/tmp"
chmod 1777 "/mnt/var/tmp"

zpool set bootfs="zroot/ROOT/default" "zroot"

mkdir -p "/mnt/boot/zfs"

zpool set cachefile="/mnt/boot/zfs/zpool.cache" "zroot"
zfs set "canmount=noauto" "zroot/ROOT/default"

export BSDINSTALL_CHROOT=/mnt
export DISTRIBUTIONS="kernel.txz base.txz"
export BSDINSTALL_DISTDIR=/mnt/usr/freebsd-dist
export BSDINSTALL_DISTSITE=https://download.freebsd.org/ftp/releases/amd64/14.1-RELEASE/
mkdir -p /mnt/usr/freebsd-dist

dhclient hn0

bsdinstall distfetch
##bsdinstall checksum
bsdinstall distextract

echo "zfs_load=\"YES\"" >> "/mnt/boot/loader.conf"
echo "zfs_enable=\"YES\"" >> "/mnt/etc/rc.conf"
echo "kern.geom.label.disk_ident.enable=\"0\"" >> "/mnt/boot/loader.conf"
echo "kern.geom.label.gptid.enable=\"0\"" >> "/mnt/boot/loader.conf"
echo "vfs.zfs.min_auto_ashift=12" >> "/mnt/etc/sysctl.conf"

FSTAB_FMT="%s\t\t%s\t%s\t%s\t\t%s\t%s\n"
printf "$FSTAB_FMT" "# Device" "Mountpoint" "FStype" "Options" "Dump" "Pass#" >> "/mnt/etc/fstab"
printf "$FSTAB_FMT" "/dev/da0p2" "none" "swap" "sw" "0" "0" >> "/mnt/etc/fstab"
printf "$FSTAB_FMT" "/dev/gpt/efiboot0" "/boot/efi" "msdosfs" "rw" "2" "2" >> "/mnt/etc/fstab"

Repeat the installation for the da0p3 (UFS) without the zfs part. and change the fstab for the UFS. It can be done via multiuser or livecd.
### UFS
# /dev/da0p3              /       ufs     rw      1       1

mount -t msdosfs /dev/da0p1 /mnt/boot/efi
mkdir -p /mnt/boot/efi/EFI/BOOT
cp /mnt/boot/loader.efi /mnt/boot/efi/EFI/BOOT/BOOTX64.efi

#list the boot entries and remove previous FreeBSD in this case it was 0003
efibootmgr
efibootmgr -B -b 0003
efibootmgr --create --activate --label "FreeBSD" --loader "/mnt/boot/efi/EFI/BOOTX64.efi"

/mnt/etc/rc.conf
hostname="test"

To boot from ZFS partition
set currdev="zfs:zroot/ROOT/default:"

To boot from UFS
set currdev=disk0p3:

----
for recovery mount under livecd
-----
mkdir /tmp/root
zpool import -fR /tmp/root zroot
zfs mount zroot/ROOT/default
ee /tmp/root/boot/loader.conf
zpool export zroot
shutdown -p now
 
vstemen said:
Maybe I should upgrade to freebsd 14.1 for further testing. Is it possible that these problems have been fixed since 14.0?
Click to expand...
vstemen said:
It's not in my loader.efi manual on freebsd-14.0. You must be on a newer release. loader.env is probably not implemented yet then.
Click to expand...
I discovered loader.env when I was on 13.x, putting in use since then when needed.

I also did some testing on 14.0 and 14.1 in VMs, but was unable to reproduce the issue.

Setup 1
  1. ZFS install first, at end of disk
  2. UFS after efi, swap, before ZFS
  3. ZFS boots by default
  4. loader.env currdev=<UFSdisk> boots fine
  5. empty loader.env boot defaults to ZFS
Setup 2
  1. UFS install first, at end of disk
  2. ZFS after efi, swap, before UFS
  3. ZFS boots by default
  4. loader.env currdev=<UFSdisk> boots fine
  5. empty loader.env boot defaults to ZFS
EDIT: In "Setup 1" root-on-ZFS was installed by the guided installer menu. See comment #21.

I didn't bother in "Setup 2" creating ZFS child datasets, only one parent dataset, "zroot", containing the whole file system.
 
Usually I ugrep(1) through man pages. By the time writing the comment 14.0 was running. Not finding anything made me believe there is no documentation, and it didn't occur to me to check out superior versions.

Thanks for pointing out the upgraded documentation Cath O'Deray, Erichans.
 
VladiBG said:
Here's the entire setup if someone want to try it. It's EFI only.
Click to expand...
You went through a lot of lengthy manual configuration, nice piece of work by the way, but that can be done faster and much easier.

First, "Auto (ZFS)" install from the guided root-on-ZFS partioning menu. From the "ZFS Configuration" menu configure a large swap in size of "swap + UFS partition", after ZFS installation finishes, still in the installer, delete swap, create a smaller swap, create UFS partition, install and configure UFS.

Exccept the manuall swap and UFS partitioning and UFS installation, configuration, all other steps are done automatic. ESP creation, ZFS pool creation, child dataset creation, ZFS property settings, UEFI menu entry, etc. The system configuration is comfortable done menu guieded.
 
You must log in or register to reply here.
Back
Top