14.0 release zfs features - gotcha

rdunkle

rdunkle

I updated from 13.2 to 14.0 release today.
used git to populate the src tree:

Code: 
git clone -b releng/14.0 --depth 1 https://git.freebsd.org/src.git /usr/src
After the installation to 14.0 updated the ports
updated the zpool
Code: 
zpool upgrade -a
When I went to reboot the machine was hung.
This was a remote system on the other side of the world.
Had to find somebody to hook up monitor and keyboard to see the boot error
Code: 
ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2
I used a another machine to create a mfsbsd bootstick with 14-stable
Booted the hung machine with the mfsbsd stick
Then ran:
Code: 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
Rebooted and that fixed the problem
 
rdunkle said:
updated the zpool
Click to expand...
Upgrade your bootcode before upgrading the pool you're booting from. Take note how you're booting though. Look at sysctl machdep.bootmethod to see what the current boot is. UEFI boot requires a different way of updating.

14.0-RELEASE has a newer version of OpenZFS than 13.2. Old bootcode, from 12.x or 13.x, will fail to boot if it hasn't been updated accordingly.
 
rdunkle said:
During the update used the gptzfsboot from the /boot. Perhaps that file is stale.
Click to expand...
When exactly? You may have done it at a stage when /boot/* hasn't been updated yet? Don't need to do this during the FreeBSD version upgrade in any case. Just don't upgrade your zpool yet. Do this once you are satisfied everything works correctly. There is no immediate need to upgrade your zpools. As long as you don't upgrade the zpool the old bootcode will continue to work fine.
 
SirDice said:
Upgrade your bootcode before upgrading the pool you're booting from. Take note how you're booting though. Look at sysctl machdep.bootmethod to see what the current boot is. UEFI boot requires a different way of updating.
Click to expand...

Which is? I have a 14.0 UEFI box that's looking for vdev_zaops_v2 and I'm stuck
 
forums.freebsd.org

Update of the bootcodes for a GPT scheme (x64 architecture)

This how-to is about the update of the bootcodes on FreeBSD as it's little to no documented. It only covers the GPT scheme and is intended for people with little knowledge of FreeBSD. When to update? At each upgrade of the system whenever it's a minor or a major one. In general, the system is...
forums.freebsd.org forums.freebsd.org
 
This bit me too, but in a little different way.
Mirror pair of boot devices/root ZFS. Through the process of upgrading from 12.x through 13.x to 14 I must have updated boot code on 1 device, upgraded the pool but forgot to update boot code on the other mirror device.
And of course a reboot wound up giving this exact same thing.
Yes the bruises on my forehead have gone away.

My opinion:
One should have a bootable device of the installed version (USB/memstick) to get to a live CD session. If you don't it's a lot of work to recover. If you do, boot it, then copy the new bootcode as appropriate.
freebsd-update
pkg install -f
update bootcode on ALL bootable devices
zpool update ONLY if you are never going to boot the pool into a lower version.

Downside of NOT doing zpool update?
You can't enable new features. Is that an issue? Maybe, maybe not. In all my time of using ZFS I've never explicitly enabled a feature. Defaults have been fine for me.
 
I wonder why people still do zpool upgrade these days.
You want to enable a specific pool feature? Then enable just it.
But first understand what the feature does, if it would really benefit you, what it changes and what you need to do to enable it without pain.
 
Andriy said:
I wonder why people still do zpool upgrade these days.
You want to enable a specific pool feature? Then enable just it.
But first understand what the feature does, if it would really benefit you, what it changes and what you need to do to enable it without pain.
Click to expand...
I think a lot is lack of understanding.
Do you need to upgrade a pool before enabling a new feature?
 
Erichans cracauer@ thanks. So zpool upgrade simply updates metadata on the pool to say "this pool supports the following features by default, in their default state (enabled/disabled)". The problem becomes a pool that is upgraded, with default enabled features that the bootloader code does not know about, so the bootloader basically pukes and stops.
 
Andriy said:
I wonder why people still do zpool upgrade these days.
You want to enable a specific pool feature? Then enable just it.
But first understand what the feature does, if it would really benefit you, what it changes and what you need to do to enable it without pain.
Click to expand...
Because we are educated to update all our software.

See the message from zpool status:
Code: 
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.

If this message disappear or change in some ways, no one or few will upgrade their pools.

On the other side, for me, the real problem is that the boot loaders aren't updated automatically. And more than that, there is no even any explanation how and why to do such updates.

So we will see again and again people complaining about that.
 
Emrion said:
Because we are educated to update all our software.

See the message from zpool status:
Code: 
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.

If this message disappear or change in some ways, no one or few will upgrade their pools.

On the other side, for me, the real problem is that the boot loaders aren't updated automatically. And more than that, there is no even any explanation how and why to do such updates.

So we will see again and again people complaining about that.
Click to expand...
That message is the root of the problem I think. It basically means "If you are never going to use this pool on an older kernel, you can upgrade. If you may use on older kernels, don't upgrade"

Boot Environments complicate this a bit.
My rule has always been "If I am never going to boot this pool on an older system, do the upgrade"
That is greatly simplified of course.
 
mer said:
My rule has always been "If I am never going to boot this pool on an older system, do the upgrade"
Click to expand...
It could even give problems on "newer" systems (as compared to "older"). The key is if the set of feature flags is compatible with the OS (=ZFS instance in it). There could be a newer Linux ZFS version that does not happen to have the feature flags support that a ZFS FreeBSD already has; or vice versa of course. If you also have to deal with any non-FreeBSD *NIX system that should remain compatible I suggest considering the use of compatibility sets (as in the manual snippet quoted); see also starting in this thread here
 
  • Like
Reactions: mer
Erichans understood. My use case is simplified:
I have FreeBSD systems, that are only booting FreeBSD systems so I don't need to worry about compatibility.

But I agree that "I'm not normal" :)
 
mer said:
… I must have updated boot code on 1 device, upgraded the pool but forgot to update boot code on the other mirror device. …
Click to expand...

Also worth noting, bug 262770 – bsdinstall with EFI system partitions on multiple disks: only one ESP gains attention
 
  • Like
Reactions: mer
I agree. Pool upgrades made more sense when pool versions were linearly increasing and there was no way to mix and match.

Nowadays, the situation is more nuanced but the pool status message is as prescriptive as it used to be.
I trained myself to just ignore that status.
Maybe if it said something like "there are new features that you might be interested in, no action is required but you may want to consult zpool-features".
 
  • Like
Reactions: mer
Andriy said:
I wonder why people still do zpool upgrade these days.
Click to expand...
because $ zpool status hints people to do so and I wanted that annoying noise to vanish from status.

I shouldn't bother what status says? Hm.

Andriy said:
But first understand what the feature does, if it would really benefit you, what it changes and what you need to do to enable it without pain.
Click to expand...

I should read the relevant docs first? Which is? How would I know?

No. If such traps are set, I'll step into them sooner or later.

I love to use a stable system and at times, when feeling bold, I experiment and carefully remove cruft. As that status message was the case.

P.S.: my case was https://forums.freebsd.org/threads/...m-klarasystems-vdev_zaps_v2.93029/post-650852
 
rdunkle said:
Code: 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
Rebooted and that fixed the problem
Click to expand...
Yeah... I did something similar. I think I forgot the -b /boot/pmbr parameter. Since it was a VPS, I couldn't boot from an cd or anything else. Long story short: I bricked my server thanks to zfs upgrade.

I run on ufs now.
 
SKull said:
Yeah... I did something similar. I think I forgot the -b /boot/pmbr parameter. Since it was a VPS, I couldn't boot from an cd or anything else. Long story short: I bricked my server thanks to zfs upgrade.

I run on ufs now.
Click to expand...
pmbr has no effect on the supported zpool version. It only searches, loads and executes the content of a freebsd-boot partition (which must contain /boot/gptzfsboot in this case). In fact, only the update of the freebsd-boot partition matters.

The question is: ada0p1 is of freebsd-boot type in your system or not? It may be ada0p2 if you have also an efi partition or anything else if you did the install of FreeBSD by hand. Not to mention the different names of the disks, depending on the driver that rules them.
 
You must log in or register to reply here.
Back
Top