OpenZFS 2.3 on 14.3-STABLE: boot fails with "ZFS: unsupported feature: org.openzfs:raidz_expansion" after `zpool upgrade`

[mer]

So I booted the 15.0-PRERELEASE memstick and manually installed 15.0-PRERELEASE on a new zfs volume as an attempt to fix the issue.

Based on what was said up a couple of posts, if the image for 15.0-PRERELEASE had the correct loader, I think you should have been able to drop to a shell and just update the loader without needing to do an install.
I know I've had to do that once or twice when I forgot to update loader on a mirror pair (I remembered to do one, forgot to do the other) and the system decided to boot off the one I forgot.

 

[SirDice]

Yeah, you don't have to install FreeBSD (whatever version) to be able to update the boot loader(s). You technically don't even need to boot the FreeBSD install media, you can simply copy the files from the install media. You typically end up having to boot the install media anyway, because your system fails to boot. As long as you have the right files you can fix the FreeBSD boot loaders with any kind of OS, even that old live Linux CD you had lying around. You just have to get the proper loader.efi, gptboot and/or gptzfsboot files. The "EFI" partition is just a FAT32 filesystem, and the freebsd-boot partition doesn't have a filesystem, the contents of gptboot or gptzfsboot are directly written to it (can be done with dd(1), you don't have to use gpart(8)).

 

[ckeen]

We caught it before release.

On the other hand is shows that few people test the installer and a new installation on -current. I know I don't.

I can confirm that a new zpool created by the 15.0-PRERELEASE memstick installer boots fine, just tested it.

 

[ckeen]

Based on what was said up a couple of posts, if the image for 15.0-PRERELEASE had the correct loader, I think you should have been able to drop to a shell and just update the loader without needing to do an install.
I know I've had to do that once or twice when I forgot to update loader on a mirror pair (I remembered to do one, forgot to do the other) and the system decided to boot off the one I forgot.

Yeah, you don't have to install FreeBSD (whatever version) to be able to update the boot loader(s). You technically don't even need to boot the FreeBSD install media, you can simply copy the files from the install media. You typically end up having to boot the install media anyway, because your system fails to boot. As long as you have the right files you can fix the FreeBSD boot loaders with any kind of OS, even that old live Linux CD you had lying around. You just have to get the proper loader.efi, gptboot and/or gptzfsboot files. The "EFI" partition is just a FAT32 filesystem, and the freebsd-boot partition doesn't have a filesystem, the contents of gptboot or gptzfsboot are directly written to it (can be done with dd(1), you don't have to use gpart(8)).

The bootcode was installed and it loads fine, the problem is that it does not support the raidz_expansion feature, so if the zpool you're trying to boot from has that feature active, it won't be able to boot from there.

 

[mer]

The bootcode was installed and it loads fine, the problem is that it does not support the raidz_expansion feature, so if the zpool you're trying to boot from has that feature active, it won't be able to boot from there.

Understood; thats why I referenced indirectly what cracauer@ said in #22 about a mismatch in the 15.0-PRERELEASE kernel and 15.0-PRERELEASE loader where the kernel supports the option, the loader does not. Yes a bug, yes caught before release. I'm guessing that you did not add that feature to the new pool you created? If so, then adding it will likely get into the same problem on 15.0-PRERELEASE

 

[ckeen]

Understood; thats why I referenced indirectly what cracauer@ said in #22 about a mismatch in the 15.0-PRERELEASE kernel and 15.0-PRERELEASE loader where the kernel supports the option, the loader does not. Yes a bug, yes caught before release. I'm guessing that you did not add that feature to the new pool you created? If so, then adding it will likely get into the same problem on 15.0-PRERELEASE

I created the pool with the default settings and it's enabled, not active though.

 

[cracauer@]

I created the pool with the default settings and it's enabled, not active though.

I thought you did an actual pool expansion before the failure?

 

[Erichans]

I created the pool with the default settings and it's enabled, not active though.

Given the current semantics, that is what is supposed to happen. When an existing pool is upgraded by means of zpool-upgrade(8) and a feature is introduced that the pool didn't support until then, it gets enabled; unless directed otherwise.

When creating a pool by a ZFS version that has support for this property, it gets enabled, unless directed otherwise. When you've used the RAIDz expansion (that is: a RAIDz expansion has taken place), the property, i.e. the feature flag state, transitions to active. Obviously this is fully supported by that ZFS kernel. When the OS boots the ZFS kernel itself is not already loaded and active at boot; we there rely on the boot loader. The loader of FreeBSD 14.3-RELEASE is ZFS enabled, however, it has not been adapted—currently—to boot a pool hat has been RAIDz expanded.

It has been suggested that RAIDz expansion should not be allowed (at all) for a boot/root pool; that, in my view, is a departure from the ZFS practice until now wrt to the introduction of a new feature flag. Going by the common practice and accepting the normal state (enabled) for a supported feature flag, and therefore allowing the pool to be RAIDz expanded, the loader has to be extended in order to support booting from a RAIDz expanded pool, catering for all eventualities that may arise. That means, for example, being able to boot a pool where the RAIDz expansion process has started but has not finished.

 

[ckeen]

I thought you did an actual pool expansion before the failure?

I did, that's where I noticed I wasn't able to boot from it anymore, because the expansion feature became active.

I installed 15.0 on a new zpool I created from the bootable installer, using default options, this pool was created with the raidz_expansion feature enabled, but not active, and the boot code is able to use it for booting.