FreeBSD upgrade from 13.1-RELEASE to 13.2-RELEASE boots into single-user mode with invalid signature in boot block error

puppydog

puppydog

I tried to upgrade my server from 13.1-RELEASE to 13.2-RELEASE today and after being prompted to reboot, I can only boot into single user mode.

Commands I ran prior to and including the reboot (remotely via SSH):
Code: 
root@server:~ # freebsd-update upgrade -r 13.2-RELEASE
root@server:~ # freebsd-update install -r 13.2-RELEASE
src component not installed, skipped
Creating snapshot of existing boot environment... done.
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.
root@server:~ # reboot
Connection to server closed by remote host.

Next, on the server I'm prompted for my geli key like normal. I enter it. It shows the FreeBSD boot menu then goes through.

It fails to import the zpool and drops me into single user mode, with only error apparent: `Invalid signature in boot block: 9f0b` Running `fsck_msdosfs /dev/mfid2p1` , with or without the -f -p options, doesn't do anything but repeat the boot signature error.
CleanShot 2023-04-11 at 19.48.21@2x.png



Checking the pool with ZFS though, nothing looks out of the ordinary (I've redacted sensitive dataset names but it all looks as expected - datasets are correct and even show the correct mountpoints):

CleanShot 2023-04-11 at 19.49.04@2x.png


I tried rolling back the snapshot that was created at the beginning of the process with `zfs rollback snapshot@blahblah` but with no effect. I tried rolling back to the snapshot before that one and got a segfault:

CleanShot 2023-04-11 at 20.04.54@2x.png


And that is where I am now. I have no idea what I can do to recover without doing a complete rebuild. Any pointers would be helpful. All the disks in the server are relatively new (all purchased within the last year) and since the pool looks fine I feel like it is unlikely that a faulty disk is the problem.
 
puppyboy said:
It fails to import the zpool and drops me into single user mode
Click to expand...
It doesn't seem to fail to import the zpool, your kernel is loaded from that zpool. If it had failed to load the pool the kernel couldn't be started and you would be stuck in loader(8), not single user mode. It has a problem with filesystem inconsistencies in your efi partition. Quick fix, remark the /boot/efi entry in fstab. Then see what's going on with the efi partition when the system is properly booted again.

Not much to rollback on, only the kernel has been changed at this point. You can rename /boot/kernel.old back to /boot/kernel to restore it. Or select one of the older boot environments in the boot menu.
 
SirDice said:
It doesn't seem to fail to import the zpool, your kernel is loaded from that zpool. If it had failed to load the pool the kernel couldn't be started and you would be stuck in loader(8), not single user mode. It has a problem with filesystem inconsistencies in your efi partition. Quick fix, remark the /boot/efi entry in fstab. Then see what's going on with the efi partition when the system is properly booted again.
Click to expand...
In the first screenshot though, it says "No pools available to import." And under /data there should be several datasets present according to zfs list showing everything with its mountpoint, but the directory is empty (though maybe this is just showing the mountpoint property and not what is actually mounted?) Though maybe I'm misunderstanding what it's telling me which is perfectly possible.

I'm not sure what you mean by remark the fstab entry though? My fstab shows /boot/efi as being on /dev/mfid2p1. Trying to mount it doesn't work though (not quite seeing why that is):
CleanShot 2023-04-11 at 20.29.54@2x.png
 
puppydog said:
I'm not sure what you mean by remark the fstab entry though? My fstab shows /boot/efi as being on /dev/mfid2p1.
Click to expand...
Remark it, as in put a # on the line to disable it. A failure to load the filesystems from fstab will cause the system to stop booting and drop you in single user mode. That's what's happening now. That /boot/efi doesn't need to be mounted in order to boot the system, so you can safely disable it. It's much easier to fix any issues with the efi partition (which is what gets mounted on /boot/efi) when you have a fully booted system.
 
Oh, sorry, I've never heard that term for that before.

Change is made and server is rebooted, and yeah it looks like it's booted up.

Server is still at 13.1-RELEASE, presumably because one of the zfs rollback attempts I made succeeded. I suppose I should go ahead and run through the 13.2 upgrade again and see what happens.
 
puppydog said:
Oh, sorry, I've never heard that term for that before.
Click to expand...
Remark is a synonym of 'comment'.

puppydog said:
Server is still at 13.1-RELEASE, presumably because one of the zfs rollback attempts I made succeeded. I suppose I should go ahead and run through the 13.2 upgrade again and see what happens.
Click to expand...
Make sure it reverted to the 13.1-RELEASE-p7_2023-04-11_165865 boot environment. That's the state of the system just before you ran freebsd-update(8).
 
SirDice said:
Remark is a synonym of 'comment'.


Make sure it reverted to the 13.1-RELEASE-p7_2023-04-11_165865 boot environment. That's the state of the system just before you ran freebsd-update(8).
Click to expand...
Okay so I've successfully performed the upgrade and freebsd-version -ku shows that I am on 13.2-RELEASE.

I have no idea how to go about troubleshooting the /boot/efi problem though - or is that even necessary? Since it boots fine with the line being commented out, is it hurting anything by just leaving it that way forever? Normally I'd like to figure out what the underlying problem really is rather than using a bandaid long term, but, as the saying goes, maybe in this case it's better to let sleeping dogs lie.
 
puppydog said:
I have no idea how to go about troubleshooting the /boot/efi problem though - or is that even necessary? Since it boots fine with the line being commented out, is it hurting anything by just leaving it that way forever?
Click to expand...
What does sysctl machdep.bootmethod tell you? If it says UEFI the system boots via that efi partition, if it's BIOS then the system gets booted from a freebsd-boot partition. So it depends if you actually need that efi partition. Personally I would still try to fix the efi partition, regardless of how the system actually boots. It's not good to leave a broken boot option around.

Does this system dual boot with any other OS? If FreeBSD is the only OS I would simply format that efi partition and copy /boot/loader.efi to it. That should quickly fix things. And I would leave the fstab entry but add a noauto to it. That way it won't be automatically mounted during boot but can be easily mounted in case you need to update the boot loader.
 
SirDice said:
What does sysctl machdep.bootmethod tell you? If it says UEFI the system boots via that efi partition, if it's BIOS then the system gets booted from a freebsd-boot partition. So it depends if you actually need that efi partition. Personally I would still try to fix the efi partition, regardless of how the system actually boots. It's not good to leave a broken boot option around.

Does this system dual boot with any other OS? If FreeBSD is the only OS I would simply format that efi partition and copy /boot/loader.efi to it. That should quickly fix things. And I would leave the fstab entry but add a noauto to it. That way it won't be automatically mounted during boot but can be easily mounted in case you need to update the boot loader.
Click to expand...
The server boots with UEFI (it's an old Dell T620 that I've had for forever) and no dual boot.

I deleted and recreated the efi partition with the following (which I *think* is the correct way to have done this?)
gpart delete -i 1 mfid2
gpart add -i 1 -t efi mfid2

Not sure what to actually do with the partition now though. I can't mount it to a temporary mountpoint so I assume it must need to be formatted with a filesystem, but I thought it would need to be in a FAT format but I don't see a way to create a fat32/whatever filesystem (unless I'm just completely overlooking it). Should I try formatting it in UFS with newfs ? Or is there something different I need to do with it first?
 
The efi partition needs to be formatted with FAT. You can use newfs_msdos(8) for this. Then mount it on /boot/efi. Create the directories: mkdir -p /boot/efi/EFI/BOOT. And copy loader.efi(8): cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTx64.efi. UEFI boot looks for EFI/BOOT/BOOTx64.efi by default (if there's nothing else configured). Double check with efibootmgr(8) how, and what's being booted. You could also save the loader to EFI/FreeBSD/loader.efi and create a custom "FreeBSD" boot entry with efibootmgr(8).

 
Awesome, thanks for the assistance SirDice. I think everything is good to go - I tested by uncommenting the problematic /etc/fstab line and rebooting, and this time it came up just fine. Before I opened this thread I was moments away from wiping everything and starting from scratch, but fortunately I didn't. I'm saving a copy of this whole thread for future reference.
 
puppydog said:
I was moments away from wiping everything and starting from scratch
Click to expand...
There's rarely a need for that. Unless you really, really, buggered things up. Like I did once, an accidental rm -rf * while in the wrong directory. I though it was /usr/ports, but it was /usr. Only realized this when it already managed to remove /usr/bin, /usr/home and a few more important directories :eek:
 
You must log in or register to reply here.
Back
Top