Mounting fails after upgrading 9.3R to 10.1R

I did a binary upgrade with freebsd-update(8).

It went something like this:

# freebsd-update -r 10.1-RELEASE upgrade
# freebsd-update install

Installing updates...chflags: ///var/empty: Read-only file system
# shutdown -r now

I didn't really mind the error and just rebooted the system when freebsd-update finished.

I adjusted the zfs readonly property for /var/empty, and I guess it just resumed the previous installation? (since it didn't keep on with installing userland updates)

# zfs set readonly=off tank/var/empty
# freebsd-update install

Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.
# shutdown -r now

Now FreeBSD is unable to mount the system.

Mounting failed from zfs:tank with error 2.

I'm not sure where to go from here and I have no clue on what the problem might be, so I would need some pointers.

I booted from a FreeBSD 10.1-RELEASE image and imported the pool by

# mkdir /tmp/mnt && zpool import -R /tmp/mnt -f tank
cannot share 'tank/user/local/storage': share (1M) failed

This didn't mount datasets with legacy mount points. I needed to mount these separately.
mount -t zfs tank /tmp/[...] .

My /boot/loader.conf (part of, ZFS related)

Is there a way of mounting all datasets under one single 'altroot'? (even those with legacy mount points).
This would give me an option to chroot <path> in to the filesystem and try freebsd-update rollback to restart the upgrade from the beginning.

Now I'm able to mount all datasets under one single 'altroot' making it possible for me to chroot into the system.
zpool import couldn't handle legacy mount points, so I had to set one by zfs set mountpoint=/ tank.

I was also thinking maybe some features were missing on my pool making FreeBSD 10.1R unable to mount zfs:tank, so I upgraded it by zfs upgrade and zpool upgrade (from a FreeBSD 10.1R image).
This broked the boot process since the installed boot loader complained about missing feature flags.

I booted the FreeBSD 10.1-RELEASE image again to upgrade the bootloader
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

Maybe I'm dealing with a partial upgrade. I'm thinking that freebsd-upgrade should've upgraded the bootloader if needed?

Above just left me in the beginning again ("Mounting failed from zfs:tank with error 2"), so I've zfs rollback [...] all individual datasets from the snapshots (a state where the system was FreeBSD 9.3R).
Now I'm not able to boot at all:

ZFS: Unspoorted feature: com.delphix:embedded_data
ZFS: Unspoorted feature: com.delphix:embedded_data
ZFS: Unspoorted feature: com.delphix:embedded_data
ZFS: can't find pool by guid
ZFS: can't find pool by guid

can't load 'kernel'

I thought this was because of the 10.1R boot loader I installed earlier (the system should be FreeBSD 9.3R since I zfs rollback to a state before freebsd-upgrade.
Or maybe because of the kernel/ZFS implementation not supporting the 'embedded_data' feature.
I booted from a 9.3R image and replaced the bootloaders again. It didn't help. I'm also unable to import my pool from a 9.3R disk image (I need a 10.1R image for that)

So it seems like I'm in a FreeBSD 9.3R system were the kernel doesn't support a ZFS feature which is written to my ZFS pool. As far as I know there are no ways of downgrading zfs pools.

What options do I got, can I make a freebsd-upgrade on the pool and try to repair it?

As I see it now my only option is to connect an additional hard drive, copy the files and make a clean reinstallation.
Last edited by a moderator: