FreeBSD-12.3 internal error failed to initialize ZFS library

byrnejb · Jan 9, 2022

A. This problem only arises when the BE is not activated and is selected from the boot menu. Possibly this a bug in beadm. See PR 261130. Work-around: use bectl to create BEs or activate beadm BEs before booting.. BEs created with bectl do not exhibit this problem.

OP
I have just noticed the internal error failed to initialize ZFS library message appeared when rebooting a system into a beadm created BE. The system came up and I can see no indication of any error with the file system. But, naturally enough, I am concerned about what this means. Does anyone have any idea what is going on?

Cath O'Deray · Jan 9, 2022

This is more often a symptom of attempting to use beadm or bectl on a file system that is not ZFS.

zfs --version && mount | grep zfs && freebsd-version -kru && uname -aKU

Emrion · Jan 9, 2022

Check this: https://forums.freebsd.org/threads/why-its-important-to-upgrade-efi-boot-bootx64-efi.79503/

byrnejb · Jan 9, 2022

grahamperrin said:
This is more often a symptom of attempting to use beadm or bectl on a file system that is not ZFS.

zfs --version && mount | grep zfs && freebsd-version -kru && uname -aKU

Code:

 zfs --version
unrecognized command '--version'
usage: zfs command args ...
where 'command' is one of the following:
. . .

mount | grep -v zfs  # shorter list
devfs on /dev (devfs, local, multilabel)
fdescfs on /dev/fd (fdescfs)

freebsd-version -kru
12.2-RELEASE-p7
12.2-RELEASE-p7
12.2-RELEASE-p10

uname -aKU          
FreeBSD vhost05.hamilton.harte-lyne.ca 12.2-RELEASE-p7 FreeBSD 12.2-RELEASE-p7 GENERIC  amd64 1202000 1202000

byrnejb · Jan 9, 2022

Code:

ll /boot
total 4572
-r--r--r--  1 root  wheel    3554 Oct 11 12:03 beastie.4th
-r--r--r--  1 root  wheel    8192 Oct 11 12:03 boot
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot0
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot0sio
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot1
-r-xr-xr-x  1 root  wheel   93696 Oct 11 12:03 boot1.efi
-r--r--r--  1 root  wheel  819200 Oct 11 12:03 boot1.efifat
-r--r--r--  1 root  wheel    7680 Oct 11 12:03 boot2
. . .

Cath O'Deray · Jan 9, 2022

byrnejb said:
… rebooting a system into a beadm created BE. The system came up …

It appears that the up system is a previous environment: 12.2-RELEASE-p7 (kernel), not 12.3.

covacat · Jan 9, 2022

activate console.log from syslog.conf in that particular BE and reboot
then look in console.log (you may get a clue where the offending zfs/zpool command was invoked)

Cath O'Deray · Jan 9, 2022

grahamperrin said:
FreeBSD bug 255318 – handbook: Document how to update the bootloader

Comment 4 draws attention to efibootmgr(8).

Rewind to 2020:

byrnejb said:
… EFI in the BIOS.

Is this the same computer … I mean, is it UEFI boot, with an ESP?

covacat said:
… the offending zfs/zpool command …

Sorry: I might have thrown people off the scent by mentioning (as an aside) a command that was not mentioned by the opening poster.

As far as I can tell – Emrion, please correct me if I'm wrong:

we have no offending command
there is, more likely, a successfully upgraded operating system (on one partition) that is currently without a suitably updated loader (on a separate msdosfs EFI system partition (ESP)).

byrnejb some months have passed since I encountered anything like this scenario, but I half-suspect that if you boot UEFI, then (with or without an updated loader on the ESP):

it should be possible to use efibootmgr to manage a boot of the 12.3 environment – as a temporary workaround.

At least, that's the theory ;-) … rather than waste time with any workaround, I guess it will be simpler to do what's required: update the loader.

byrnejb · Jan 9, 2022

grahamperrin said:
It appears that the up system is a previous environment: 12.2-RELEASE-p7 (kernel), not 12.3.

The new BE will not boot.

byrnejb · Jan 11, 2022

Emrion said:
Check this: https://forums.freebsd.org/threads/why-its-important-to-upgrade-efi-boot-bootx64-efi.79503/

I am experimenting on a host that has already been updated to 12.3 and is running that version

Code:

[root@vhost01 ~ (master)]# freebsd-version
12.3-RELEASE
[root@vhost01 ~ (master)]# efibootmgr -v
efibootmgr: efi variables not supported on this system. root? kldload efirt?
[root@vhost01 ~ (master)]# kldload efirt
kldload: can't load efirt: module already loaded or in kernel

Looking back in my daybook I see that I had cause to update the boot partition during a zpool recovery. In that case my notes state that I did this: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0.

Looking at /var/run/dmesg.boot I do not find any references to EFI:

Code:

[root@vhost01 ~ (master)]# grep -i efi /var/run/dmesg.boot
[root@vhost01 ~ (master)]#

Checking on the host running 12.3 , not the one I am having trouble with, I see this:

Code:

[root@vhost01 ~ (master)]# gpart show
=>        40  5860533088  ada0  GPT  (2.7T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  5856335872     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

=>        40  5860533088  ada1  GPT  (2.7T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  5856335872     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

=>        40  5860533088  ada2  GPT  (2.7T)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  5856335872     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

On the host that I am trying to upgrade from 12.2 to 12.3 I see this:

Code:

[root@vhost05 ~ (master)]# gpart show
=>         40  15628053088  ada0  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048      4194304     2  freebsd-swap  (2.0G)
      4196352  15623856128     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada1  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048      4194304     2  freebsd-swap  (2.0G)
      4196352  15623856128     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada2  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048      4194304     2  freebsd-swap  (2.0G)
      4196352  15623856128     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada3  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
         1064          984        - free -  (492K)
         2048      4194304     2  freebsd-swap  (2.0G)
      4196352  15623856128     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

I gather from this that the host is booted from BIOS and that /boot/boot.efi is not appropriate . So I believe that I should update the boot loader with /boot/gptzfsboot instead.

Comments?

Emrion · Jan 11, 2022

That's the same problem. EFI or BIOS, you have to update the bootcodes and, in your case, especially your freebsd-boot partition with the lastest gptzfsboot.

Update of the bootcodes for a GPT scheme (x64 architecture)

This how-to is about the update of the bootcodes on FreeBSD as it's little to no documented. It only covers the GPT scheme and is intended for people with little knowledge of FreeBSD. Edit: I wrote a script that gives the right commands to type after inspecting your system and can possibly...

forums.freebsd.org

byrnejb · Jan 11, 2022

The boot loader does not seem to be the problem.

In the first instance the system boots normally. The issue only occurs when booting into a boot environment other than the default reboot.

In the second instance, updating the MBR with /boot/gptzfsboot did not change the observed behaviour. The BE still fails to boot.

Since the BE as initially created by beadm is supposed to be the equivalent of the reboot boot environment it eludes me how the first fails to start with a zfs library error whilst the second starts without a problem.

Any suggestions as to what else could cause this behaviour?

covacat · Jan 11, 2022

most likely it does not load zfs.ko or opensolaris.ko

Emrion · Jan 11, 2022

byrnejb said:
The boot loader does not seem to be the problem.

In the first instance the system boots normally. The issue only occurs when booting into a boot environment other than the default reboot.

In the second instance, updating the MBR with /boot/gptzfsboot did not change the observed behaviour. The BE still fails to boot.

Since the BE as initially created by beadm is supposed to be the equivalent of the reboot boot environment it eludes me how the first fails to start with a zfs library error whilst the second starts without a problem.

Any suggestions as to what else could cause this behaviour?

It's exactly the problem I described in my post. Maybe your're out of luck...

byrnejb · Jan 12, 2022

If I create a BE using beadm; and I activate that BE; and I reboot the system; and I do not interact with the boot process; then the system boots.

If I reboot the system from that BE; and I interact with the boot process; and I select the original BE; then the system boots.

If I activate the original BE; and reboot the system; and I do not interact with the boot menu; then the system boots.

If I reboot the system; and I select the new BE from the boot menu; then the zfs error is reported and the system fails to boot.

byrnejb · Jan 12, 2022

I worked around this by creating a BE named NEWBOOT and activating it. I then rebooted and the system came up in NEWBOOT. I updated the host to 12.3 in this BE. Once the install was finished (bootx2) I created another BE named OTHER1 and rebooted. At the boot menu I selected that BE and the system halted with the same zfs error as before.

The zfs issue arises only when the alternative BE is selected at the boot menu. The system starts but one cannot login, presumably because it cannot allocate a tty or pty. In any case, the filesystem is inaccessible.

The question arises is this a beadm problem, or is this an issue with the boot menu?

Emrion · Jan 13, 2022

Did you update the bootloader of all your disks and how?

Statement like: "it's not the bootloader" without explanation why you came to this conclusion is worthless.

byrnejb · Jan 13, 2022

I created a new BE (BECTLBOOT) using bectl and I can boot into that from the boot menu without activating the BE. Thus, it is evident that the problem lies with beadm and not with the system.

byrnejb · Jan 13, 2022

Emrion said:
Did you update the bootloader of all your disks and how?

Statement like: "it's not the bootloader" without explanation why you came to this conclusion is worthless.

Code:

gpart show | grep 'ada\|boot'
=>         40  15628053088  ada0  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
=>         40  15628053088  ada1  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
=>         40  15628053088  ada2  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)
=>         40  15628053088  ada3  GPT  (7.3T)
           40         1024     1  freebsd-boot  (512K)

ls -l /boot/gptzfsboot
-r--r--r--  1 root  wheel  158858 Jan 12 15:48 /boot/gptzfsboot

for HDD in {0..3}; do gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada${HDD}; done;
partcode written to ada0p1
bootcode written to ada0
partcode written to ada1p1
bootcode written to ada1
partcode written to ada2p1
bootcode written to ada2
partcode written to ada3p1
bootcode written to ada3

Emrion · Jan 13, 2022

Ok. You're lucky because vermaden, the author of beadm is sometimes around. He'll certainly want to help you. That said, I can't understand this problem because beadm is a sh script that just use base commands.