/boot directory deleted: How to recover with ZFS backups

While testing the capabilities of beadm(1) and boot environments on a spare machine, I deleted the /boot directory, and the system would no longer boot. With competent help from IRC, I was able to recover without booting a rescue system (except in case 3c) as defined below). I'm here to describe the procedure, because it doesn't seem to be documented anywhere. Corrections and additions are welcome.

Please note the following conditions for this mode of recovery:
1) The system must be using the new loader(8) (written in Lua), which is the default in FreeBSD 12.x and 13.x – this will not work for the older default loader (written in Forth C and Forth (thanks, SirDice)).
2) The system must be using UEFI boot – this will not work for BIOS booting (though it does work for systems installed with the "BIOS+UEFI" option, as long as UEFI boot is active)
3) You must have some ZFS-based backup of the dataset containing /boot. This could be:
3a) a beadm(1)- or bectl(8)-administered backup boot environment
3b) a manually created clone of your zpool/ROOT/default (or equivalent dataset) (but see the warning for the final step if you don't have beadm(1) installed)
3c) a manually taken snapshot of your zpool/ROOT/default (or equivalent dataset) (but you'll need a rescue system, and see the warning for the final step if you don't have beadm(1) installed)
4) Your system can be fully encrypted with geli or not, this works both ways. Just enter your passphrase whenever the bootloader asks for it.


So you've deleted /boot, and your system won't boot. Now what?

First, you'll find yourself dropped to a boot console with a prompt like this.
Code:
Type '?' for a list of commands, 'help' for more detailed help.
OK
If you don't, or if your prompt says Shell> instead, try rebooting and watch for the following message:
Code:
Failed to find bootable partition
press any key to interrupt reboot in 5 seconds
… and press any key.

At the OK prompt, we first need to specify a backup dataset with an intact /boot directory. Let's say that zpool/ROOT/backup exists, whether that's a beadm(1)-created boot environment, or a clone you created manually.

(If you don't have a clone, but just a snapshot, you have to do the following first (I'm just giving you the rough steps, this is well-documented elsewhere): boot a live system, geli attach your drive if it's encrypted, zpool import -o altroot=... your pool, and finally clone the snapshot like zfs clone zroot/ROOT/default@snap zroot/ROOT/backup. Now you can reboot and follow this guide.)

At the OK prompt, enter the following command: set currdev=zfs:zroot/ROOT/backup:
Take a good look at the syntax, as it's very unusual. If it fails, you've probably just typed it wrong. Pay attention to the spelling of currdev and the final :.

If you've done it correctly, you'll just get the next OK .
Next, enter the following command: include /boot/lua/loader.lua.
If all is well, the good old Beastie bootloader screen should come up.

Now, you're not quite out of the woods, but close:
- At the Beastie screen, press 8 for the boot environment selection. Then press 2 until the backup boot environment is selected.
- Use 1 to get back to the main menu, and press 2 for single user boot. (You can try a default multi user boot, but I've seen problems with devfs(8) missing, and single user mode is enough for the last few steps.)
- Just hit Enter on the prompt that says: Enter full pathname of shell or RETURN for /bin/sh:
- In the shell, run: beadm activate backup and reboot. (If you don't have beadm(1) installed, you may use bectl(8) or try a multi user boot in the last step to install it with pkg install beadm. If that fails, there is certainly a way to register your backup clone dataset as the active one for the next reboot, but I don't know the exact steps. At least you're inside your system in single user mode, you'll surely be able to get the rest of the way out of the hole.)
- Breathe a sigh of relief when your system reboots normally :)
 
Last edited:
If 3) is true (fileset with proper /boot exists somewhere on the ZFS pool) you can intervene with the bootloader as I mentioned few years ago in this post. That's valid for non-uefi boots.
 
  • Like
Reactions: mtu
In the meantime, I also learned that bectl(8) has been in base since at least 12.0, with similar (or identical?) functionality to beadm(1). So that might help anyone stuck in single-user mode without network to install sysutils/beadm.
 
Back
Top