zfsboot fails with "unsupported feature" after upgrade to 13.2-RELEASE-p1_2023-08-04 and zpool upgrade

LudwigP · Oct 23, 2023

I have a system originally built as a FreeBSD 13.1R with root ZFS, which I've upgraded using freeebsd-update to 13.2R-p1. I've upgraded the zpool and enabled all features, and after a reboot, am unable to boot it. I have tried updating the bootcode using all methods I found reference to, but it hasn't helped.

This appears on the console when I boot from the hard drive(s):

Code:

BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
ZFS: unsupported feature: com.delphix.redaction_bookmarks
ZFS: pool zroot is not supported

Can't find /boot/zfsloader

Can't find /boot/loader

Can't find /boot/kernel/kernel

FreeBSD/x86 boot
Default: /boot/kernel/kernel
boot:

I am able to access the ZFS pool by booting from a USB stick imaged with FreeBSD-13.2-STABLE-amd64-20231005-2464d8c5e296-256505-mini-memstick.img.xz and selecting Shell

Code:

# dhclient re0
# mkdir /tmp/ssh
# mount_unionfs /tmp/ssh /etc/ssh
# printf "PermitRootLogin yes\nPasswordAuthentication yes\nPermitEmptyPasswords yes\nUsePAM no\no" >> /etc/ssh/sshd_config
# /etc/rc.d/sshd onestart

#{SSH in for ease of copy-paste.}

root@:~ # geom concat load
root@:~ # geom mirror load
root@:~ # mkdir /tmp/root
root@:~ # zpool import -a -R /tmp/root
root@:~ # zpool list -v
NAME              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot            1.81T   584G  1.24T        -         -     1%    31%  1.00x    ONLINE  /tmp/root
  mirror-0       1.81T   584G  1.24T        -         -     1%  31.4%      -    ONLINE
    ada2p4       1.82T      -      -        -         -      -      -      -    ONLINE
    concat/zfs1  1.82T      -      -        -         -      -      -      -    ONLINE
logs                 -      -      -        -         -      -      -      -  -
  gpt/slog         16G   424K  15.5G        -         -     0%  0.00%      -    ONLINE
cache                -      -      -        -         -      -      -      -  -
  gpt/l2arc       127G   127G   143M        -         -     0%  99.9%      -    ONLINE

root@:~ # zfs list
NAME                                           USED  AVAIL     REFER  MOUNTPOINT
zroot                                          584G  1.19T       96K  /tmp/root/zroot
zroot/ROOT                                    24.4G  1.19T       96K  none
zroot/ROOT/13.1-RELEASE-p7_2023-07-15_171848     8K  1.19T     4.22G  /tmp/root
zroot/ROOT/13.1-RELEASE_2023-03-05_000007        8K  1.19T     1.35G  /tmp/root
zroot/ROOT/13.2-RELEASE-p1_2023-07-15_172053     8K  1.19T     4.29G  /tmp/root
zroot/ROOT/13.2-RELEASE-p1_2023-08-04_232738     8K  1.19T     19.8G  /tmp/root
zroot/ROOT/default                            24.4G  1.19T     18.7G  /tmp/root
zroot/tmp                                     2.37G  1.19T     2.37G  /tmp/root/tmp
zroot/usr                                      557G  1.19T       96K  /tmp/root/usr
zroot/usr/home                                 387M  1.19T      346M  /tmp/root/usr/home
zroot/usr/local                                556G  1.19T       96K  none
zroot/usr/local/share                          556G  1.19T      556G  none
zroot/usr/ports                                810M  1.19T      810M  /tmp/root/usr/ports
zroot/usr/src                                   96K  1.19T       96K  /tmp/root/usr/src
zroot/var                                     2.88M  1.19T       96K  /tmp/root/var
zroot/var/audit                                 96K  1.19T       96K  /tmp/root/var/audit
zroot/var/crash                                 96K  1.19T       96K  /tmp/root/var/crash
zroot/var/log                                 1.88M  1.19T      888K  /tmp/root/var/log
zroot/var/mail                                 508K  1.19T      256K  /tmp/root/var/mail
zroot/var/tmp                                  224K  1.19T       96K  /tmp/root/var/tmp
root@:~ # zfs mount zroot/ROOT/default

root@:~ # ls -l /boot/pmbr /boot/gptzfsboot /tmp/root/boot/gptzfsboot /tmp/root/boot/pmbr
-r--r--r--  1 root  wheel  156046 Oct  5 09:41 /boot/gptzfsboot
-r--r--r--  1 root  wheel     512 Oct  5 09:41 /boot/pmbr
-r--r--r--  1 root  wheel  156502 Jul 16 00:18 /tmp/root/boot/gptzfsboot
-r--r--r--  1 root  wheel     512 May 12  2022 /tmp/root/boot/pmbr

root@:~ # egrep 'ada.*device|ada.*sectors' /var/run/dmesg.boot
ada0: <ST1000DM003-1ER162 CC45> ACS-2 ATA SATA 3.x device
ada0: 953869MB (1953525168 512 byte sectors)
ada1: <Samsung SSD 860 EVO 250GB RVT01B6Q> ACS-4 ATA SATA 3.x device
ada1: 238475MB (488397168 512 byte sectors)
ada2: <ST2000DM001-1ER164 CC26> ACS-2 ATA SATA 3.x device
ada2: 1907729MB (3907029168 512 byte sectors)
ada3: <ST1000DM003-1ER162 CC43> ACS-2 ATA SATA 3.x device
ada3: 953869MB (1953525168 512 byte sectors)

ada0 and ada1 are on an extra SATA controller
ada2 and ada3 are on the chipset SATA controller.

I boot non-UEFI, GPT. The BIOS is UEFI capable but I'm pretty sure I have always booted non-UEFI, and in the course of trying to fix this issue, I messed up the EFI partition in following some forum posts trying to update it. So I can probably fix and attempt EFI boot if someone gives me the correct steps.

My annotations below are in {}.

I have concat'ed partitions of the 2 1 TB disks together to be the mirror of the 2TB drive, and I have an addition SSD to act as L2ARC and SLOG (just to play with the features). Doing the "geom load concat" causes some of the ada[0-3] names to be unusable for gpart, so I have to use the diskid label instead.

Code:

root@:~ # gpart show diskid/DISK-Z4Y9GBS5 {ada0}
=>        40  1953525088  diskid/DISK-Z4Y9GBS5  GPT  (932G)
          40        4056                        - free -  (2.0M)
        4096  1953517568                     1  freebsd-zfs  (932G) {label zfs1a}
  1953521664        3464                        - free -  (1.7M)

root@:~ # gpart show ada1
=>       40  488397088  ada1  GPT  (233G)
         40       4056        - free -  (2.0M)
       4096   33554432     1  freebsd-zfs  (16G) {label slog}
   33558528  266338304     2  freebsd-zfs  (127G) {label l2arc}
  299896832  188500296        - free -  (90G)

root@:~ # gpart show ada2 {diskid/DISK-Z4Z3YP4B}
=>        40  3907029088  ada2  GPT  (1.8T)
          40      532480     1  efi  (260M) {label efiboot0}
      532520        1024     2  freebsd-boot  (512K) {label gptboot0}
      533544         984        - free -  (492K)
      534528     4194304     3  freebsd-swap  (2.0G) {label swap0}
     4728832  3902300160     4  freebsd-zfs  (1.8T) {label zfs0}
  3907028992         136        - free -  (68K)

root@:~ # gpart show diskid/DISK-Z4Y0PJNM (ada3)
=>        40  1953525088  diskid/DISK-Z4Y0PJNM  GPT  (932G)
          40      532480                     1  efi  (260M) {label efiboot1}
      532520        1024                     2  freebsd-boot  (512K) {label gptboot1}
      533544         984                        - free -  (492K)
      534528     4194304                     3  freebsd-swap  (2.0G) {label swap1}
     4728832  1948794880                     4  freebsd-zfs  (929G) {label zfs1b}
  1953523712        1416                        - free -  (708K)

root@:~ # gconcat list
Geom name: zfs1
State: UP
Status: Total=2, Online=2
Type: AUTOMATIC
ID: 2074157860
Providers:
1. Name: concat/zfs1
   Mediasize: 1997983972352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
Consumers:
1. Name: diskid/DISK-Z4Y0PJNMp4
   Mediasize: 997782978560 (929G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   Start: 1000200994304
   End: 1997983972352
2. Name: diskid/DISK-Z4Y9GBS5p1
   Mediasize: 1000200994816 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   Start: 0
   End: 1000200994304

root@:~ # gpart bootcode -b /boot/pmbr  diskid/DISK-Z4Y9GBS5
bootcode written to diskid/DISK-Z4Y9GBS5
root@:~ # gpart bootcode -b /boot/pmbr ada1
bootcode written to ada1
root@:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada2
partcode written to ada2p2
bootcode written to ada2
root@:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 diskid/DISK-Z4Y0PJNM
partcode written to diskid/DISK-Z4Y0PJNMp2
bootcode written to diskid/DISK-Z4Y0PJNM

After reboot, same result. I tried updating bootcode using the on-disk boot bootcodes in zroot/ROOT/default:

Code:

root@:~ # gpart bootcode -b /tmp/root/boot/pmbr diskid/DISK-Z4Y9GBS5 # {ada0}
bootcode written to diskid/DISK-Z4Y9GBS5
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr ada1
bootcode written to ada1
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr -p /tmp/root/boot/gptzfsboot -i 2  diskid/DISK-Z4Z3YP4B # {ada2}
partcode written to diskid/DISK-Z4Z3YP4Bp2
bootcode written to diskid/DISK-Z4Z3YP4B
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr -p /tmp/root/boot/gptzfsboot -i 2 diskid/DISK-Z4Y0PJNM # {ada3}
partcode written to diskid/DISK-Z4Y0PJNMp2
bootcode written to diskid/DISK-Z4Y0PJNM

Same result. Then I tried using a different ROOT:

In place of

Code:

root@:~ # zfs mount zroot/ROOT/default

I executed

Code:

root@:~ # zfs mount zroot/ROOT/13.2-RELEASE-p1_2023-08-04_232738

Bootcode appears the same as zroot/ROOT/default:

Code:

root@:~ # ls -l /boot/pmbr /boot/gptzfsboot /tmp/root/boot/gptzfsboot /tmp/root/boot/pmbr
-r--r--r--  1 root  wheel  156046 Oct  5 09:41 /boot/gptzfsboot
-r--r--r--  1 root  wheel     512 Oct  5 09:41 /boot/pmbr
-r--r--r--  1 root  wheel  156502 Jul 16 00:18 /tmp/root/boot/gptzfsboot
-r--r--r--  1 root  wheel     512 May 12  2022 /tmp/root/boot/pmbr

Repeated:

Code:

root@:~ # gpart bootcode -b /tmp/root/boot/pmbr diskid/DISK-Z4Y9GBS5 # {ada0}
bootcode written to diskid/DISK-Z4Y9GBS5
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr ada1
bootcode written to ada1
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr -p /tmp/root/boot/gptzfsboot -i 2  diskid/DISK-Z4Z3YP4B # {ada2}
partcode written to diskid/DISK-Z4Z3YP4Bp2
bootcode written to diskid/DISK-Z4Z3YP4B
root@:~ # gpart bootcode -b /tmp/root/boot/pmbr -p /tmp/root/boot/gptzfsboot -i 2 diskid/DISK-Z4Y0PJNM # {ada3}
partcode written to diskid/DISK-Z4Y0PJNMp2
bootcode written to diskid/DISK-Z4Y0PJNM

Same result.

I don't understand why none of the bootcode supports this ZFS feature considering I'm using the ZFS version from this FreeBSD version. Any ideas on getting this system bootable again?

SirDice · Oct 23, 2023

LudwigP said:
I don't understand why none of the bootcode supports this ZFS feature considering I'm using the ZFS version from this FreeBSD version.

I'm guessing you are UEFI booting the machine and you didn't update loader.efi in the EFI partition.

LudwigP · Oct 23, 2023

SirDice said:
I'm guessing you are UEFI booting the machine and you didn't update loader.efi in the EFI partition.

I addressed this already. I am not booting with EFI.

SirDice · Oct 23, 2023

LudwigP said:
I've upgraded the zpool and enabled all features

How did you do this? Because com.delphix.redaction_bookmarks should be added in 13.x, the ZFS of 12.x doesn't know it though. It almost looks like it's using a gptzfsboot(8) from 12.x, not 13.x.

I suggest booting the 13.2 install media and writing the bootcode with the gptzfsboot(8) that's on the install media.

LudwigP · Oct 23, 2023

SirDice said:
How did you do this? Because com.delphix.redaction_bookmarks should be added in 13.x, the ZFS of 12.x doesn't know it though. It almost looks like it's using a gptzfsboot(8) from 12.x, not 13.x.

I suggest booting the 13.2 install media and writing the bootcode with the gptzfsboot(8) that's on the install media.

I used zpool upgrade.

And I already booted off a 13.2-STABLE memstick and copied its bootcode to the disks. Your last two posts brought up things I have already addressed.

SirDice · Oct 23, 2023

Hmm. Don't really know what's happening. But I may have a working theory: The system boots from ada3, it has bootcode so that should be fine. gptzfsboot(8) then starts looking for a suitable filesystem to boot. It finds the freebsd-zfs partition (label zfs1b) on ada3 first, but that's not a 'complete' volume. It's half of a gconcat(8) volume. And bombs out trying to read it.

Can you play around with the boot order settings in the BIOS? Set it to boot from ada2 instead of ada3?

LudwigP · Oct 24, 2023

I do have the BIOS set to boot ada2. I have tried booting all drives via the BIOS boot menu. ada2 and ada3 show the output from my original post. ada0 and ada1 show "Missing boot loader". UEFI off ada2 results in a blank screen and back to the boot loader because, as I said in my first post, I messed up my EFI file system.

And your explanation for not liking the half-volume doesn't work with the fact that I used to be able to boot before doing my freebsd-update and zpool upgrades. It can load *a* zfs boot loader, but not one that likes my current pool.

Anyways, I tried fixing my EFI boot volume since my BIOS is EFI capable. This is, again, booting off a USB drive written with FreeBSD-13.2-STABLE-amd64-20231005-2464d8c5e296-256505-mini-memstick.img.xz.

Code:

root@:~ # gpart show -l diskid/DISK-Z4Z3YP4B # {ada2}
=>        40  3907029088  diskid/DISK-Z4Z3YP4B  GPT  (1.8T)
          40      532480                     1  efiboot0  (260M)
      532520        1024                     2  gptboot0  (512K)
      533544         984                        - free -  (492K)
      534528     4194304                     3  swap0  (2.0G)
     4728832  3902300160                     4  zfs0  (1.8T)
  3907028992         136                        - free -  (68K)

root@:~ # gpart show -l diskid/DISK-Z4Y0PJNM # {ada3}
=>        40  1953525088  diskid/DISK-Z4Y0PJNM  GPT  (932G)
          40      532480                     1  efiboot1  (260M)
      532520        1024                     2  gptboot1  (512K)
      533544         984                        - free -  (492K)
      534528     4194304                     3  swap1  (2.0G)
     4728832  1948794880                     4  zfs1b  (929G)
  1953523712        1416                        - free -  (708K)

root@:~ # mount_msdosfs /dev/diskid/DISK-Z4Z3YP4Bp1 /tmp/efi0
root@:~ # mount_msdosfs /dev/diskid/DISK-Z4Y0PJNMp1 /tmp/efi1
mount_msdosfs: /dev/diskid/DISK-Z4Y0PJNMp1: Invalid argument
root@:~ # newfs_msdos /dev/diskid/DISK-Z4Y0PJNMp1
/dev/diskid/DISK-Z4Y0PJNMp1: 532288 sectors in 16634 FAT16 clusters (16384 bytes/cluster)
BytesPerSec=512 SecPerClust=32 ResSectors=1 FATs=2 RootDirEnts=512 Media=0xf0 FATsecs=65 SecPerTrack=63 Heads=16 HiddenSecs=0 HugeSectors=532480
root@:~ # mount_msdosfs /dev/diskid/DISK-Z4Y0PJNMp1 /tmp/efi1
root@:~ # ls -l /boot/loader.efi
-r-xr-xr-x  2 root  wheel  906240 Oct  5 09:41 /boot/loader.efi
root@:~ # cp /boot/loader.efi /tmp/efi0/efi/boot/bootx64.efi
root@:~ # cp /boot/loader.efi /tmp/efi0/efi/freebsd/loader.efi
root@:~ # cp /boot/loader.efi /tmp/efi1/efi/boot/bootx64.efi
root@:~ # cp /boot/loader.efi /tmp/efi1/efi/freebsd/loader.efi
root@:~ # efibootmgr -v
Boot to FW : false
BootCurrent: 0004
Timeout    : 1 seconds
BootOrder  : 0002, 0003, 0004, 0001
 Boot0002* Hard Drive  BBS(HD,,0x0)
 Boot0003* USB  BBS(USB,,0x0)
+Boot0004* UEFI:  USB Flash Memory1.04 PciRoot(0x0)/Pci(0x10,0x0)/USB(0x3,0x0)/HD(1,MBR,0x90909090,0x1,0x10418)
                                      VenHw(2d6447ef-3bc9-41a0-ac19-4d51d01b4ce6,2000550053004200200046006c0061007300680020004d0065006d006f007200790031002e00300034000000)
 Boot0001* UEFI OS HD(1,GPT,8fecec8d-ba38-11ed-8af3-d05099aa3161,0x28,0x82000)/File(\EFI\BOOT\BOOTX64.EFI)
                      diskid/DISK-Z4Z3YP4Bp1:/EFI/BOOT/BOOTX64.EFI /tmp/efi0//EFI/BOOT/BOOTX64.EFI

After selecting EFI boot at the BIOS boot menu:

(Please excuse typos and the abbreviations of device IDs. This came from iOS OCR off a photo, manually corrected)

Code:

Consoles: EFI console
ZFS: unsupported feature: com.delphix:redaction_bookmarks
ZFS: pool zroot is not supported
   Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk3p1:
FreeBSD/amd64 EFI loader, Revision 1.1

    Command line arguments: loader.efi
    Image base: 0x89ff7000
    EFI version: 2.31
    EFI Firmware: American Megatrends (rev 4.654)
    Console: efi (0)
    Load Path: \EFI\BOOT\BOOTX64.EFI
    Load Device: PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x0,0x0)/HD(1,GPT,8fecec8d-ba38-11ed-8af3-d05099aa3161,0x28,...
    BootCurrent: 0001
    BootOrder: 0002 0003 0004 0001[*] 0005
    Boot Info Path: HD(1,GPT,8fecec8d-ba38-11ed-8af3-d05099aa3161,0x28,0x82000)/\EFI\BOOT\BOOTX64.EFI\
Ignoring Boot0001: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(Ox11,0x0)/Sata(0x0,0x0)/HD(1,GPT,8fecec8d-ba38-...
Setting currdev to disk3p1:
Trying: PciRoot (0x0)/Pci(0x11,0x0)/Sata(0x0,0x0)/HD(2,GPT,90154297-ba38-...
Setting currdev to disk3p2:
Trying: PciRoot (0x0)/Pci(0x11,0x0)/Sata(0x0,0x0)/HD(3,GPT,9055610b-ba38-...
Setting currdev to disk3p3:
Trying: PciRoot (0x0)/Pci(0x11,0x0)/Sata(0x0,0x0)/HD(4,GPT,907b7355-ba38-...
Setting currdev to
Failed to find bootable partition
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.


Туре '?' for a list of commands, 'help' for more detailed help.
OK

LudwigP · Oct 29, 2023

SOLVED

So, the problem turned out to be the specific ZFS feature, not that I updated my FreeBSD. After trying to install the bootloader from a FreeBSD 14.0-RC3 memstick and having it fail the same way, I repeated some googling, and it turns out the problem was that the com.delphix:redaction_bookmarks ZFS feature is not read-only compatible, and the ZFS on Root Bootloader will refuse to load it.

This bug report told me what was up: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264174

After deleting my redaction bookmark and all of my snapshots (deleting snapshots was probably unnecessary), I was able to again boot my system.

zfsboot fails with "unsupported feature" after upgrade to 13.2-RELEASE-p1_2023-08-04 and zpool upgrade

LudwigP

SirDice

Administrator

LudwigP

SirDice

Administrator

LudwigP

SirDice

Administrator

LudwigP

LudwigP