System won't boot after power outage - "can't load 'kernel'" (12.0 RELEASE amd64)

Julf said:
I now have the ZFS system mounted on /tmp/tmproot
Click to expand...
That shows the pool itself and the data it contains is still good. It'll be easier to work from here. The loader(8) is quite versatile but a bit of a pain to use interactively when trying to solve issues.

Can you show the output from gpart show?
 
SirDice said:
Can you show the output from gpart show?
Click to expand...

Currently running stand-alone, single user, without network, so can't cut&paste, but I hope the attached pic is readable.
 

Attachments

  • IMG_20191113_174647244.jpg
    IMG_20191113_174647244.jpg
    206.3 KB · Views: 136
This is slightly dangerous but at worst the system won't boot, which it doesn't do now anyway, so the risk is minimal. Make sure the installer image you used is the same version as your system was (similar enough, it doesn't need to have the exact same patch level, just the same major version).

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Then reboot, cross your fingers and hope that was enough to get it booting again.
 
SirDice said:
This is slightly dangerous but at worst the system won't boot, which it doesn't do now anyway, so the risk is minimal. Make sure the installer image you used is the same version as your system was (similar enough, it doesn't need to have the exact same patch level, just the same major version).

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Then reboot, cross your fingers and hope that was enough to get it booting again.
Click to expand...

Before I do that, is there any way to check what version the system was?
 
Julf said:
There is no file called "kernel" the boot/kernel directory of the imported filesystem.
Click to expand...
That would definitely explain the "can't load kernel" messages. Is the entire directory empty or are the *.ko kernel modules still there? If everything is mostly there but you're only missing kernel you can try copying the kernel from the install media you used. Yes, that would be a blank (patch-less) version but something is better than nothing. Alternatively, if you're still unsure about the version, copy the kernel file from the kernel.old directory.
 
SirDice said:
That would definitely explain the "can't load kernel" messages. Is the entire directory empty or are the *.ko kernel modules still there? If everything is mostly there but you're only missing kernel you can try copying the kernel from the install media you used. Yes, that would be a blank (patch-less) version but something is better than nothing.
Click to expand...

Yes, all the .ko files are there.
 
Unfortunately copying the kernel didn't yet solve the problem. Now it seems the issue is inability to mount root. Of course the error message is just above the last console screen - is there any way to pause/scroll back boot console output?
 
Julf said:
Unfortunately copying the kernel didn't yet solve the problem. Now it seems the issue is inability to mount root.
Click to expand...
Well, it's some progress, now the kernel has started.

Julf said:
Of course the error message is just above the last console screen - is there any way to pause/scroll back boot console output?
Click to expand...
Standard FreeBSD console allows scrolling if you hit the "Scroll lock" button on your keyboard. You can use the cursor keys or PgUp/PgDown to scroll up and down. Hit "Scroll lock" again to stop. That key might be tricky on a laptop, you may need to use some function (Fn) key for it.
 
SirDice said:
Standard FreeBSD console allows scrolling if you hit the "Scroll lock" button on your keyboard. You can use the cursor keys or PgUp/PgDown to scroll up and down. Hit "Scroll lock" again to stop. That key might be tricky on a laptop, you may need to use some function (Fn) key for it.
Click to expand...

Thanks. Had to swap to a proper full keyboard instead of the slimline one I was using. What I get is:

Trying to mount root from zfs:zroot/ROOT/default []...
Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system.
 
Is there a safe way to reinstall the system without overwriting existing application data (such as the dovecot mailboxes)?
 
Which kernel did you copy? The error looks like it has problems loading the zfs.ko kernel module. Which may be because the kernel you copied and the modules don't exactly match up. You probably need to copy a whole /boot/kernel/ directory, modules and all.

Julf said:
Is there a safe way to reinstall the system without overwriting existing application data (such as the dovecot mailboxes)?
Click to expand...
Probably not necessary to do a complete reinstall. We just need to get the kernel in order, then we can boot the system further. On the installation media there's a /usr/freebsd-dist/kernel.txz, you can extract that one; tar -C /your/tmp/root -zxvf /usr/freebsd-dist/kernel.txz. Change the /your/tmp/root to where you imported your pool to. Unpacking those files is what the installer normally does.
 
SirDice said:
Which kernel did you copy? The error looks like it has problems loading the zfs.ko kernel module. Which may be because the kernel you copied and the modules don't exactly match up. You probably need to copy a whole /boot/kernel/ directory, modules and all.


Probably not necessary to do a complete reinstall. We just need to get the kernel in order, then we can boot the system further. On the installation media there's a /usr/freebsd-dist/kernel.txz, you can extract that one; tar -C /your/tmp/root -zxvf /usr/freebsd-dist/kernel.txz. Change the /your/tmp/root to where you imported your pool to. Unpacking those files is what the installer normally does.
Click to expand...

That worked! System boots OK. Many thanks!
 
Hurray! Good. Keep in mind that you now have a "plain" 12.0 kernel, no security patches. You may want to try running freebsd-update(8), hopefully that detects the "old" kernel and will update it for you.

As you're here now, I often copy that "plain" kernel from the install media to a /boot/kernel.RELEASE/ or /boot/kernel.GOOD/. You can just copy it, then you can use it in emergencies like this one.
 
SirDice said:
Hurray! Good. Keep in mind that you now have a "plain" 12.0 kernel, no security patches. You may want to try running freebsd-update(8), hopefully that detects the "old" kernel and will update it for you.
Click to expand...

Done - just about to see if it still boots. :)

As you're here now, I often copy that "plain" kernel from the install media to a /boot/kernel.RELEASE/ or /boot/kernel.GOOD/. You can just copy it, then you can use it in emergencies like this one.
Click to expand...

Excellent advice!
 
Still, the question is, how could the system end up without any kernel. Sure, a simple power outage won't do that. ZFS is extremely safe in that sense.
Neither will it upgrade automatically; and when you upgrade it will prompt for reboot a couple of times. Further, ZFS pool will NEVER be upgraded without asking that EXPLICITLY and your answering in affirmative. So you still need to recall the chain of events, just to make sure this doesn't happen again.
I remember having that problem of system not finding /boot/kernel/kernel due to loader's weird ideas about where to look for boot/kernel/kernel. Don't know the mechanics behind it, but what helped me then was adding kern.bootfile=/boot/kernel/kernel line to /etc/sysctl.conf.
 
Last edited:
free-and-bsd said:
Still, the question is, how could the system end up without any kernel.
Click to expand...
We can only speculate, but I'm guessing a rm -r /boot/kernel/* (or similar) happened at some point, and it went unnoticed until the power outage.

I know I've accidentally rm'ed a few files in the past, only to discover problems related to their absence weeks later. Fortunately they weren't that important, and I had backups, but yes, he should be snapshotting his whole tree periodically. Rolling snapshots can save you lots of headaches in the future.
 
I will definitely snapshot from here on. Pretty sure I didn't remove the kernel myself, still not sure what happened.
 
You must log in or register to reply here.
Back
Top