ZFS ZFS boot loader "ZFS: can't find pool by guid" during 11.2 upgrade

(Not sure if this would be better on the Installation and Upgrade section, but it maybe seems more storage related. Mods please feel free to move.)

During the reboot as part of an upgrade from FreeBSD 11.1 to 11.2, got an error from the ZFS enabled boot loader saying "can't find 'kernel'". Other 11.1 to 11.2 upgrades, including ZFS systems went fine, though they may have had different partitions, etc.

Attached are some screenshots of the original message, lsdev, and gpart show. (Pardon not being able to grab console text as text.) Why would ZFS lose track of guids, and how can I fix it?
 

Attachments

  • zfs-confused1.PNG
    zfs-confused1.PNG
    19.5 KB · Views: 439
  • zfs-confused2.PNG
    zfs-confused2.PNG
    21.2 KB · Views: 476
  • livecd-gpart-show1.PNG
    livecd-gpart-show1.PNG
    38.1 KB · Views: 355
  • livecd-gpart-show2.PNG
    livecd-gpart-show2.PNG
    46.4 KB · Views: 383
  • livecd-gpart-show3.PNG
    livecd-gpart-show3.PNG
    30.2 KB · Views: 421
Last edited:
Here's a screenshot of zpool import.
 

Attachments

  • livecd-zpool-import.PNG
    livecd-zpool-import.PNG
    63.5 KB · Views: 430
jeffc Were you able to resolve this? Same problem on an old Celeron dual core that requires MBR (BIOS) configuration. The boot has to be there (the system is running) - but I'm lost as to where to discover 'boot' went.
 
This fixed it for me:

zpool status
pool: zroot
state: ONLINE​
scan: none requested​
config:​
NAME STATE READ WRITE CKSUM​
zroot ONLINE 0 0 0​
ada0s1d ONLINE 0 0 0​
errors: No known data errors​

# sudo zpool import -f bootpool
# zfs list

NAME USED AVAIL REFER MOUNTPOINT
bootpool 246M 1.51G 245M /bootpool
zroot 3.34G 173G 88K /zroot
zroot/ROOT 666M 173G 88K none
zroot/ROOT/default 666M 173G 666M /
zroot/tmp 120K 173G 120K /tmp
zroot/usr 2.68G 173G 88K /usrs
zroot/usr/home 156K 173G 156K /usr/home
zroot/usr/ports 1.41G 173G 1.41G /usr/ports
zroot/usr/src 1.27G 173G 1.27G /usr/src
zroot/var 612K 173G 88K /var
zroot/var/audit 88K 173G 88K /var/audit
zroot/var/crash 88K 173G 88K /var/crash
zroot/var/log 168K 173G 168K /var/log
zroot/var/mail 92K 173G 92K /var/mail
zroot/var/tmp 88K 173G 88K /var/tmp
 
Generally speaking these issues are most likely to come from upgrading your pool using # zpool upgrade without re-installing the bootcode using gpart.
 
Thanks guys, both of those make sense. Hopefully the above may help some folks, so thanks for sharing.

It does feel like the boot code got out of sync with my ZFS file system as a result of the FreeBSD upgrade process. I ended up reinstalling FreeBSD and using the FreeBSD installer to set up ZFS. It did the right thing with the boot partition and bootcode of course. I then restored a ZFS snapshot of the old system onto the resulting ZFS filesystem. Everything works normally.

I had not done a zpool upgrade before trying to do the FreeBSD upgrade, but probably should have. I did end up successfully doing a zpool upgrade later and also writing the current bootcode to the boot partition.
 
As a related aside, I tried to zpool upgrade a different system (also upgraded to FreeBSD 11.2), and also write the current bootcode to the boot partition, but the resulting system also failed to boot.

Here's the gpart statements used to create a related raidz1 zpool (they're functionally identical to what I used on the other system that also failed to boot after the upgrades):
Code:
gpart destroy -F mfid0
gpart destroy -F mfid1
gpart destroy -F mfid2
gpart create -s gpt mfid0
gpart create -s gpt mfid1
gpart create -s gpt mfid2
gpart add -s 222 -a 4k -t freebsd-boot -l boot0 mfid0
gpart add -s 222 -a 4k -t freebsd-boot -l boot1 mfid1
gpart add -s 222 -a 4k -t freebsd-boot -l boot2 mfid2
gpart add -s 8g -a 4k -t freebsd-swap -l swap0 mfid0
gpart add -s 8g -a 4k -t freebsd-swap -l swap1 mfid1
gpart add -s 8g -a 4k -t freebsd-swap -l swap2 mfid2
gpart add -a 4k -t freebsd-zfs -l disk0 mfid0
gpart add -a 4k -t freebsd-zfs -l disk1 mfid1
gpart add -a 4k -t freebsd-zfs -l disk2 mfid2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid2

This was working fine until the FreeBSD 11.2 and zpool upgrades.

I did expand the boot partitions to 512k so the current bootcode could fit by taking some space from the swap partitions. Maybe the expand didn't work right? Here are the steps I used to expand the boot partition and write what I assume was the current boot code from FreeBSD 11.2:

Code:
# gpart show
=>        34  5859372989  mfid0  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40         216      1  freebsd-boot  (108K)
         256    16777216      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)

=>        34  5859372989  mfid1  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40         216      1  freebsd-boot  (108K)
         256    16777216      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)

=>        34  5859372989  mfid2  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40         216      1  freebsd-boot  (108K)
         256    16777216      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0
gpart: /dev/mfid0p1: not enough space
# gpart delete -i 1 mfid0
mfid0p1 deleted
# gpart delete -i 1 mfid1
mfid1p1 deleted
# gpart delete -i 1 mfid2
mfid2p1 deleted
# swapoff /dev/mfid0p2
# swapoff /dev/mfid1p2
# swapoff /dev/mfid2p2
# gpart delete -i 2 mfid0
mfid0p2 deleted
# gpart delete -i 2 mfid1
mfid1p2 deleted
# gpart delete -i 2 mfid2
mfid2p2 deleted
# gpart add -s 512k -a 4k -t freebsd-boot -l boot0 mfid0
mfid0p1 added
# gpart add -s 512k -a 4k -t freebsd-boot -l boot0 mfid1
mfid1p1 added
# gpart add -s 512k -a 4k -t freebsd-boot -l boot0 mfid2
mfid2p1 added
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0
partcode written to mfid0p1
bootcode written to mfid0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid1
partcode written to mfid1p1
bootcode written to mfid1
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid2
partcode written to mfid2p1
bootcode written to mfid2
# gpart add -a 4k -t freebsd-swap -l swap0 mfid0
mfid0p2 added
# gpart add -a 4k -t freebsd-swap -l swap0 mfid1
mfid1p2 added
# gpart add -a 4k -t freebsd-swap -l swap0 mfid2
mfid2p2 added
# swapon /dev/mfid0p2
# swapon /dev/mfid1p2
# swapon /dev/mfid2p2
# swapinfo /dev/mfid0p2
Device          1K-blocks     Used    Avail Capacity
/dev/mfid0p2      8388204        0  8388204     0%
/dev/mfid1p2      8388204        0  8388204     0%
/dev/mfid2p2      8388204        0  8388204     0%
Total            25164612        0 25164612     0%
# gpart show
=>        34  5859372989  mfid0  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40        1024      1  freebsd-boot  (512K)
        1064    16776408      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)

=>        34  5859372989  mfid1  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40        1024      1  freebsd-boot  (512K)
        1064    16776408      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)

=>        34  5859372989  mfid2  GPT  (2.7T)
          34           6         - free -  (3.0K)
          40        1024      1  freebsd-boot  (512K)
        1064    16776408      2  freebsd-swap  (8.0G)
    16777472  5842595544      3  freebsd-zfs  (2.7T)
  5859373016           7         - free -  (3.5K)
Does anyone see anything obviously wrong with the above for recent ZFS, etc.?

We have not been using an explicit -b numeric parameter for gpart add -t freebsd-boot . Is that needed now? It looks like the boot partition starts at 40, which I assume is ok.
 
Last edited:
-a 1M will give you a little extra space for the future.

The recommended size for the boot part is now 512K. Some of the GPT boot loaders are over 128 KB in size and done are passing 256 KB.
 
-a 1M will give you a little extra space for the future.

The recommended size for the boot part is now 512K. Some of the GPT boot loaders are over 128 KB in size and done are passing 256 KB.
Thanks much! I did use the recommended 512K size. Interesting that 1M might be better for future expansion.

Also, to be clear the second system above also failed to boot. Would definitely like to learn how to fix this. None of the drives use ZFS encryption.
 
hello jeffc did you find a solution to this? - i have a similar experience going from 11.2 -> 11.3. and are wondering what would be the solution for this.
 
Hi ehasting, I ended up doing a fresh ZFS install, then a ZFS snapshot restore onto the new partitions created by the default install. I was not able to recover the broken system.

Question for all: is 512K for the boot partition still enough or are we pushing that limit and should be using 1M now?
 
Back
Top