Solved zpool feature@bookmark_v2 remains 'active' after all bookmarks are destroyed, unbootable system

janprzy

janprzy

My FreeBSD 13.0 machine just failed to boot stating that my zroot has com.datto:bookmark_v2 enabled, which is not supported. I booted a live USB and destroyed all bookmarks. zfs list -t bookmark does not return anything anymore, so I assume they are all actually gone.
However, the pool's feature@bookmark_v2 property remains 'active' when, according to the zpool-features(5) manpage, it should revert to 'enabled' after all bookmarks have been destroyed. My system still does not boot.

Does anyone know what to do? Any suggestions would be appreciated.
 
janprzy said:
it should revert to 'enabled' after all bookmarks have been destroyed.
Click to expand...
If all bookmarks are truly removed and that feature is not set back to the enabled state it may be a bug.
Have you tried booting from the live usb and then using zpool set feature@bookmark_v2=disabled on your pool?

Edit:
Interesting that it's failing, I'm assuming that your zroot was running FreeBSD13 prior to this, with the bookmarks created? Trying to figure out how it got to that state.
 
I have tried that, it doesn't work. The feature cannot be disabled once it has been enabled.

One theory is that v1 and v2 bookmarks are handled separately and I only deleted v1 bookmarks. When I manually create a bookmark, feature@bookmarks switches to 'active', when I destroy that bookmark, it reverts to 'enabled', so that manually created bookmark was likely v1.

However, I have no idea how to specifically manage v2 bookmarks and couldn't find anything about that.
 
That doesn't work either, unfortunately. 'bookmark_v2' is not a valid filesystem type, and zfs list -t all -r zroot | grep "#" yields no results.
 
I'm out of ideas and looking at man pages isn't given me any ideas.

If you do
zpool get all zroot | grep -i book

What does feature@bookmark_written say? man zpool-features implies it's tied to both bookmarks and bookmarks_v2 and may give a clue. my understanding of bookmarks_v2 from the man page is they are created when you do a bookmark but it seems to relate to size so is done automatically.

as root, do
zpool history zroot

and look for clues there. I've not done anything with bookmarks on my systems, but I see snapshot commands in the history. It may let you figure out a name so you can do a zfs destroy.
 
janprzy said:
However, I have no idea how to specifically manage v2 bookmarks and couldn't find anything about that.
Click to expand...
Well, how did you create them? Is the pool from another machine/OS/person?
Maybe if you import it on a (live) system that explicitly supports the v2 booksmars feature, you'll get somewhere.
 
zpool get all zroot | grep -i book and zpool history zroot are not helpful. The bookmarks where created automatically by zrepl.

The live system I'm using is the official FreeBSD 13.0 iso, I assume that supports v2 bookmarks
 
The zpool-features(5) manpage from 13.0-RELEASE confirms much of what you suspected:
Code: 
       bookmark_v2

       GUID              com.datto:bookmark_v2
       READ-ONLY COMPATIBLE      no
       DEPENDENCIES          bookmark, extensible_dataset

       This feature enables the creation and management  of larger book-
       marks which are needed for other features in ZFS.

       This feature becomes active when a v2 bookmark is created and will
       be returned to the enabled state when all v2 bookmarks are de-
       stroyed.
 
I'm not sure, shouldn't the bootcode have been updated when I went to 13.0? How can I check this?
 
if you efi boot you can probably md5 compare loader.efi
or what is the exact name
otherwhise dd the zfsboot partition code and compare
dd if=/some/bootpart of=/tmp/test.boot bs=1 count=the_size_of_boot_code_shipped_with_13
then md5 compare
 
covacat said:
if you efi boot you can probably md5 compare loader.efi
Click to expand...
YES! Copying /boot/loader.efi from the live system to /efi/boot/BOOTx64.efi on my EFI partition fixed it! I don't know why my bootloader was outdated, but the problem is solved now. Thank you all for your help!
 
janprzy said:
YES! Copying /boot/loader.efi from the live system to /efi/boot/BOOTx64.efi on my EFI partition fixed it! I don't know why my bootloader was outdated, but the problem is solved now. Thank you all for your help!
Click to expand...
Many users have been struggling with this upon updating to 13.0. Updating the boot code isn't part of the usual freebsd-update routine, and requires manual intervention. In this case, since the ZFS code has changed dramatically with 13.0, many people are running into trouble with their zpools when the bootcode isn't updated.

Personally, I think it's FreeBSD's fault for allowing many users to run into this problem. The general consensus on these Forums, however, is that the users are to blame for not reading the documentation properly. Make up your own mind ;)
 
I had the same problem, but compounded by an inability to mount the efi msdos filesystem from a recovery USB stick. gpart bootcode didn't have any effect.

In the end I deleted the msdos partition and re-created it, then copied in the directory structure from the USB stick's msdos partition.

gpart didn't show any errors but apparently silently failed to update the bootcode? That's a new one for me.
 
mtu said:
Many users have been struggling with this upon updating to 13.0. Updating the boot code isn't part of the usual freebsd-update routine, and requires manual intervention. In this case, since the ZFS code has changed dramatically with 13.0, many people are running into trouble with their zpools when the bootcode isn't updated.

Personally, I think it's FreeBSD's fault for allowing many users to run into this problem. The general consensus on these Forums, however, is that the users are to blame for not reading the documentation properly. Make up your own mind ;)
Click to expand...
I was lucky to stumble across vermaden's excellent post a couple of days ago, and re-wrote my upgrade plan for FreeBSD 13.
Non-ZFS file systems generally don't change all that often, so the driver for reading form disk in the boot code tends to be fairly static.
But ZFS is complex and under constant development, so it's fairly obvious, when you reflect on it, that the boot code may need updating quite regularly.
It's not mentioned in the Handbook, and I'm thinking that there is a lack of prominent documentation regarding how and when to update the bootcode.
However there is some good reading on the subject in the EXAMPLES section of gpart(8).
 
You must log in or register to reply here.
Back
Top