Solved Recover from broken upgrade, no GELI access

[Torsten]

Hi. After some hours trying around I am quite lost now. Maybe someone can give me a hint.
I wanted to upgrade my laptop system from 11.1-RELEASE-p11 to 11.2-RELEASE. As I have successfully used boot environments on a virtual machine, I first set up beadm:

portmaster sysutils/beadm
beadm list
beadm create 2018-08-06-00
beadm list

Everything seemed fine. But I felt too safe, so I didn’t try a reboot to see if I can really select one of those two boot environments. Omitting this was a mistake, I guess. Then I started the upgrade with

freebsd-update -r 11.2-RELEASE upgrade

It took several attempts, but finally went trough just fine. I don’t know if it is normal that some drivers get broken at this point. My wireless stopped working after I restarted wpa_supplicant. Anyway, I rebooted.
So, here are the problems: No boot environments other than “default” can be selected. Where is my other boot environment? When I try to boot normally, then at some early point the machine reboots. When I try to boot with the old kernel, then I only get a single user mode shell. The contents of /var/log/messages is the following:

…
kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
kernel: ada0: <SanDisk …> … SATA 3.x device
kernel: ada0: Serial Number …
kernel: ada0: 600.000MB/s …
kernel: ada0: Command Queueing enabled
kernel: ada0: 915715MB (… 512 byte sectors)
…
kernel: Trying to mount root from zfs:zroot/ROOT/default []...
…
kernel: GEOM_ELI: Device ada0p5.eli created.
kernel: GEOM_ELI: Encryption: AES-XTS 256
kernel: GEOM_ELI:     Crypto: hardware
kernel: pid 85 (zfs), uid 0: exited on signal 6 (core dumped)
kernel: pid 90 (zfs), uid 0: exited on signal 6 (core dumped)
kernel: pid 91 (zfs), uid 0: exited on signal 6 (core dumped)
…
init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
…

I try some stuff to see, what’s going on:

# mount
zroot/ROOT/default on / (zfs, …)
devfs on /dev (…)
procfs on /proc (…)

# geli list
Geom name: ada0p5.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: hardware
Version: 7
UsedKey: 0
Flags: BOOT
KeysAllocated: 223
KeysTotal: 223
Providers:
1. Name: ada0p5.eli
   Mediasize: … (890G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: ada0p5
   Mediasize: … (890G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2097152
   Mode: r1w1e1
Geom name: ada0p4.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN
KeysAllocated: 1
KeysTotal: 1
Providers:
…
# geli attach /dev/ada0p5
Enter passphrase:
geli: Wrong key for ada0p5
# geli attach ada0p5
Enter passphrase:
geli: Wrong key for ada0p5
# zfs list
internal error: Invalid argument
Abort (core dumped)
# zfs mount zroot/ROOT/default
internal error: Invalid argument
Abort (core dumped)
# zpool import -f -R /tmp/mount zroot
internal error: Invalid argument
Abort (core dumped)

I tried all this stuff because I also booted an 11.2-Install-USB-Stick and unsuccessfully tried to access my encrypted ZFS. geli always complains “Wrong key for ada0p5”, despite the key is certainly correct. When I boot from my encrypted SSD, then the correct key is accepted (and there I also see that ada0p5 is the correct partition name), but after that I end up in single user mode and my home directory is not mounted and I cannot see all my files either.
This all seems like a mess to me. I would like to go in any of these directions:

• roll back to my saved boot environment
• boot with the new or the old kernel
• access my files in single user mode
• access my files on the live system
• or whatever

For the moment I have no other idea than a complete new installation and restore from backup (which would not be so cool—much work, some data loss). Do you have any ideas?

Edit 1: I saw that the reboot problem may be linked to Virtualbox, which I indeed have installed as well. But I have no clue how I could remove these lines in /boot/loader.conf as I do not know how to access that file.

Edit 2: Before everything broke I also ran freebsd-update install, of course. That one took several attempts and finally succeeded. The upgrade command alone does nothing harmful, I guess.

Edit 3: This goes out from my once again running system. Yeay! I learned a lot, and especially that: Besides the encrypted zpool “zroot” there is an unencrypted zpool “bootpool”. I didn’t know that, but that explains, how I can access the loader.conf without password in the boot process. Ok. So in the running live system, I had to

zpool import -f -R /tmp/mntboot bootpool

Then I could

vi /tmp/mntboot/bootpool/boot/loader.conf

and comment out the beforementioned line

vboxdrv_load="YES"

Furthermore the geli attach had not worked, because I had to name the extra key file that’s residing in the bootpool as well. So now I can

geli attach -k /tmp/mntboot/bootpool/boot/encryption.key ada0p5

and then

zpool import -f -R /tmp/mntroot zroot

Or I simply boot into single-user mode of my installed system and

mount -uw /

to be able to edit /etc/rc.conf, where I also comment out the line

vboxnet_enable="YES"

Thanks for accompanying me in learning all this stuff. Now I only have to finish my upgrade to 11.2, fix that VirtualBox shit and dive into the boot environments topic again to see why all that didn’t work as expected.

P.S.: That VirtualBox shit should really be mentioned in the release notes. Seems to be a known problem.

P.P.S.: It would be really nice if this forum would not block Tor. This is stupid.

 

[gqgunhed]

You really saved my day. My server crashed the same way because of the "vboxdrv" and "vboxnet" being enabled.
It took me a little different road to get my zpool imported, but basically your hint was the solution.

# Download a FreeBSD-live-memstick.img
# boot with live system and enter shell
mkdir /tmp/mnt

# mount geli encrypted volumes, otherwise you will get an IOError trying to import rpool
# I have a mirrored ZFS pool on drives ada2 and ada3
geli attach /dev/ada2p3
geli attach /dev/ada3p3

# import zpool to another location
zpool import -o altroot=/tmp/mnt
# "/" of rpool needs to be manually mounted
zfs mount rpool/ROOT/default

# make the corrections
cd /tmp/mnt
vi ./boot/loader.conf   # comment out vboxdrv_load="YES"
vi ./etc/rc.conf        # comment out vboxnet_enable="YES"

zpool export
# reboot