Upgrade 9.3 > 10.0: Can't boot ZFS root

[marvel]

Hi,

Updated 9.3 > 10.0 with freebsd-update install. It told me to reboot first to load the new kernel, then run the command again. However it's not booting, the error I get is:

ZFS: unsupported feature: com.delphix:hole_birth

I booted from 10.0-RELEASE ISO image, tried to mount ZFS but same error appeared. So I downloaded a 10.1 PRERELEASE ISO, booted from that and now I can mount my ZFS volume.

From the suggestions that told me would fix it I typed:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

and to the mirror:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

However, on boot the error remains and my system is unusable.

Any idea?

Thanks!

 

[marvel]

I found the cause. I did a fresh install of 9.3 back then which supports newer ZFS functions than 10.0-RELEASE does. So I guess I'm screwed unless I update to 10.1-PRERELEASE or 11.0 which is not really an option. I probably can do a rollback to 9.3 but I don't want to since I already reinstalled all of my ports it's going to be a mess anyway.

So I'm just going to backup, re-format and install 10.0-RELEASE from scratch. When 10.1 becomes available I should have no problems upgrading then.

Seems a bit weird though to add new ZFS functions to 9.3 and still allow upgrading from 9.3 to 10.0-RELEASE while it doesn't support these features.

 

[SirDice]

Seems a bit weird though to add new ZFS functions to 9.3 and still allow upgrade from 9.3 to 10.0-RELEASE while it doesn't support these features.

The reason for the difference is because 9.3 was released after 10.0. As 10.1 is now at the pre-release stage it shouldn't be long until there's a 10.1-RC. The release candidates are "official" versions that can be upgraded with freebsd-update(8). The 10.1-PRERELEASE is actually 10-STABLE, which can't be updated with freebsd-update(8).

 

[marvel]

Seems a bit weird though to add new ZFS functions to 9.3 and still allow upgrade from 9.3 to 10.0-RELEASE while it doesn't support these features.

The reason for the difference is because 9.3 was released after 10.0. As 10.1 is now at the pre-release stage it shouldn't be long until there's a 10.1-RC. The release candidates are "official" versions that can be upgraded with freebsd-update(8). The 10.1-PRERELEASE is actually 10-STABLE, which can't be updated with freebsd-update(8).

I understand but don't you agree it's weird to give 9.3 a ZFS version that the 10.0 kernel can't handle? I know root on ZFS is expirimental, but a warning when you try to upgrade 9.3 to 10.0 would've been nice

 

[SirDice]

I understand but don't you agree it's weird to give 9.3 a ZFS version that the 10.0 kernel can't handle?

It's not that weird once you understand how releases are made. New features (like those new ones in ZFS) are tested on -CURRENT. At some point those changes are Merged From Current (MFC). The changes will then find it's way to every -STABLE version, like 9-STABLE and 10-STABLE. The RELEASE versions are created from -STABLE.

So the "history" is something like this:

• 9.2-RELEASE came out
• 10.0-RELEASE came out
• new feature added to -CURRENT
• MFC the new features to 9-STABLE and 10-STABLE
• 9.3-RELEASE came out
• 10.1-RELEASE

 

[marvel]

Yeah it makes sense but it still sucks that I had to destroy my zroot. If I started with 9.2 I wouldn't have any problems going to 9.3 and then 10.0. It's just that I created my initial ZFS Root with 9.3.

I know now and I won't make this mistake again xD

 

[SirDice]

If I started with 9.2 I wouldn't have any problems going to 9.3 and then 10.0.

You would have if you did a zpool upgrade on 9.3.

I know now and I won't make this mistake again

Yeah, it was probably better to wait until 10.1 is released.

 

[kpa]

Release versions of FreeBSD are strictly "feature locked" meaning no new features will be added to them under any circumstances. This means that there won't be any updates to 10.0-RELEASE to make it support ZFS features that were added to releases made later than itself such as 9.3-RELEASE. Even adding a warning to 10.0-RELEASE (other than in the online errata) that it will not support ZFS pools made with 9.3-RELEASE is unlikely to happen because that would mean changing the behaviour of an already released system.

 

[marvel]

Release versions of FreeBSD are strictly "feature locked" meaning no new features will be added to them under any circumstances. This means that there won't be any updates to 10.0-RELEASE to make it support ZFS features that were added to releases made later than itself such as 9.3-RELEASE. Even adding a warning to 10.0-RELEASE (other than in the online errata) that it will not support ZFS pools made with 9.3-RELEASE is unlikely to happen because that would mean changing the behaviour of an already released system.

Yes I understand but I still think this is different since it only affects RootOnZFS which is experimental and imho still subject to change.

If I go 9.2 > 9.3 > 10.0 I wouldn't have had any issues since my ZFS version would be from 9.2 and 10.0 is a later release.
If I go 9.3 > 10.0 then I will have issues.

I don't think you can explain that to a normal person. It just makes no sense to upgrade the ZFS version on 9.3 and make it newer than the one in 10.0 so you can't mount a ZFS made in 9.3 in 10.0. That's a big flaw. At least the new features should be disabled so it keeps compatibility with 10.0.

But oh well, it is what it is and people are not gonna going to change this because of me complaining about it so I'll have peace with it I'll just make sure to never upgrade to a release that came out earlier than the one I'm running.

Thanks for replying.

 

[SirDice]

I don't think you can explain that to a normal person. It just makes no sense to upgrade the ZFS version on 9.3 and make it newer than the one in 10.0 so you can't mount a ZFS made in 9.3 in 10.0. That's a big flaw.

You can, if you explain that 9.3 was released after 10.0. Counter-intuitive I know but that's just the way FreeBSD works. It's simply because 9 and 10 are on different and separate paths.

 

[phoenix]

If I go 9.2 > 9.3 > 10.0 I wouldn't have had any issues since my ZFS version would be from 9.2 and 10.0 is a later release.
If I go 9.3 > 10.0 then I will have issues.

That's not true.

If you start with 9.2 and create the ZFS pool, then upgrade to 9.3, anytime you run "zpool status" you'll get a message that you aren't running the latest version of ZFS and that you should upgrade your pool. If you upgrade your pool with any of the new features, then you will not be able to import that pool on any systems that don't support those features (aka, 10.0).

It's not the FreeBSD version that matters. It doesn't matter which version of ZFS was used to create the pool. All that matters is which features you've enabled on the pool, and what the last version of ZFS was used to upgrade it/enable features.

The trick is to make a note of which ZFS features you've enabled on a pool, and to check whether or not future versions of the zpool command can import pools with those features enabled.

It's also why you need to think about what kinds of OSes you'll be using to import the pool in the future when you create the pool. Do you specify version 28 so that you can import the pool using Solaris/OpenSolaris and it's derivatives (ie, maximum compatibility)? Do you specify version 5000 so that's it importable across OpenZFS-using OSes (like FreeBSD, FreeNAS, PC-BSD, Illumos, Delphix, etc)? If you go with version 5000, which specific features do you enable? For root pools, you might not need/want all the fancy features.

This is also why you need to look into Boot Environments. Clone the root filesystem, mark it live, reboot into the clone. Do the upgrade. If anything fails, you just boot back to the original filesystem. No rebuilding of the pool required. And you just wait for 10.1 to be released before trying the upgrade again in a new boot environment.