Solved The quest for unencrypted /boot and ZFS native encrypted /

ivanov

ivanov

Hello Free Beastie Besties,

I have a two zfs pool root-on-ZFS Debian 12.4 setup: 1G unecrypted `bpool` and 100G zfs native encrypted `rpool .
(I got those names from following the framework in this guide: https://openzfs.github.io/openzfs-docs/Getting Started/Debian/Debian Bullseye Root on ZFS.html )

I'd like to add FreeBSD to the mix and install it into the same ZFS pools. The end goal for me is to be able to dual boot FreeBSD and Debian. It would be nice to share /home across the two operating systems, but not a dealbreaker so long as I can get access to say /debian/home/username while in FreeBSD, and vice versa, I can manage some symlinks so that the home away from home still feels appropriately cozy.

From information I've gathered it seems that this is possible, but I'm not super clear on the details. I've so far verified that I am able to boot into FreeBSD 14.0 installation media (via Ventoy with an assist from this thread https://github.com/ventoy/Ventoy/issues/2636 issuecomment -1858203099 ), and I am able to `zpool import -f` both pools (after `zfs load-key rpool`) under FreeBSD running from the installation disk.

Next I used the shell mode partitioning, created the FreeBSD filesystem datasets (eschewing the `/default` convention and using `bpool/BOOT/freebsd` for `/boot` the rest on the encrypted zpool). I ended up this these filesystem datasets:

bpool/BOOT/debian
bpool/BOOT/freebsd
rpool/BOOT/
rpool/ROOT/debian
rpool/ROOT/freebsd
rpool/fbsd/tmp
rpool/fbsd/usr
rpool/fbsd/usr/home
rpool/fbsd/usr/obj
rpool/fbsd/usr/ports
rpool/fbsd/usr/ports/distfiles
rpool/fbsd/usr/ports/packages
rpool/fbsd/usr/src
rpool/fbsd/var
rpool/fbsd/var/audit
rpool/fbsd/var/crash
rpool/fbsd/var/log
rpool/fbsd/var/mail
rpool/fbsd/var/tmp
rpool/...# other debian mount points
Click to expand...

populated `/tmp/bsdinstall_etc/fstab`, and mounted the file systems under `/mnt`, and ran the rest of the installation. But this is where things got a bit fuzzy, and I just started trying stuff. (Most other people describing encrypted root-on-ZFS under FreeBSD were using GELI, but that's not what I'm trying to do here)

I saw mention somewhere that to accomplish dual-booting Debian 12.4 / FreeBSD 14.0, that I need to use all legacy mountpoints for the OS directories ( in other words the ones in `heir(7)` ), and that I would use `/etc/fstab`
to specify what gets mounted where. If that's correct, then I'm not clear on how I can get to `/etc/fstab` if root is encrypted. It seems like I want to get to a state similar to how I boot into Debian where I boot using a dataset on the unencryted `bpool`, get a prompt for unlocking the key of the encrypted zpool, and carry on booting the system from there.

I copied the freebsd bootloader to the EFI partition, and perhaps not too unsuprisingly, it fails to boot. At least not without some manual intervention. First I end up in the EFI loader

Ignoring Boot0004: Only one DP found
Trying ESP: PciRoot(0x0).../HD(2,GPT...)
Setting curdev to disk0p2
Trying: PciRoot(0x0).../HD(3,GPT...)
Setting curdev to zfs:bpool/BOOT/freebsd:
Trying: PciRoot(0x0).../HD(4,GPT...)
Setting curdev to zfs:rpool:
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
Click to expand...

The zio_read/ZFS errors are repeated a few times, with "Failed to find bootable partition" inserted in there, and ending with a "ERROR: cannot open /boot/lua/loader.lua: no such file or directory".

I can `set currdev="zfs:bpool/BOOT/freebsd:"`, and then have tried `load kernel/kernel` followed by `boot`, or also
sprinkling in `load kernel/zfs.ko` just in case, but end up with a system that looks like its starting to boot, but then
gives up and just reboots itself.

Sure would appreciate some guidance on how to complete this quest. Thank you!
 
angry_vincent said:
FreeBSD bootloader has no support for booting from native encrypted ZFS pool.
Click to expand...
I have a small unencrypted ZFS pool (bpool) to hold the contents of /boot.

Edit: I realize that my thread title might have thrown you off, since it was previously "root on native encrypted ZFS", so I've updated the title to better reflect what I'm seeking, thanks.
 
grahamperrin said:

263171 – add loader(8) and boot loader menu support for boot with OpenZFS-encrypted ROOT

bugs.freebsd.org bugs.freebsd.org
Click to expand...
Thank you, I did come across that one, but I thought that bug was to be able to boot into ZFS native encrypted root without any workarounds.
I am operating under the possibly false impression that there is currently a way to have an unencrypted /boot that gets you enough of a system so you can zfs load-key to decrypt the fs datasets for the remaining mountpoints.

I lead myself to believe this might be possible, based on how it works for me in Debian, where bootloader, kernel, and initrd are on an unencrypted /boot, and even found a description suggestive of a similar approach possibly working on FreeBSD, for example this post by free-and-bsd :
free-and-bsd said:
Well in that case maybe try this.
Create a small UFS partition and place /boot/kernel/{kernel zfs.ko opensolaris.ko linux.ko & other modules you use } along with /boot/{ loader zfsloader & other loader stuff } in there. Among them /boot/loader.conf with that line about vfs.root.mountfrom.
Because whatever you need to mount root from encrypted zpool (kernel + zfs modules) will all be there in that unencrypted UFS partition.

...Finally, use freebsd-boot partition and legacy loader to boot stuff from your UFS partition. Which will proceed to mount root from your encrypted zpool. Good old method.
EDIT: (1) for that matter, even EFI loader can be used to boot stuff from that UFS partition pretty much the same way, I guess.
Click to expand...
free-and-bsd said:
4) If you consider using ZFS encryption in your zpool, GRUB2 will have problems booting directly from it. You'll need to use a non-encrypted /boot partition with kernel/modules. You can use full-disk ZFS on one of your disks... though honestly, I've never noticed that much of an advantage in it. So finally I switched back to GPT scheme. Then again, your /boot partition can be anywhere (except for Windows partition LOL).
Click to expand...
But maybe these were just theories that don't work out in practice?
 
Apologies for misreading. I'll strike through.

… an unencrypted /boot that gets you enough of a system so you can zfs load-key …
Click to expand...

If I understand correctly, an unencrypted /boot/ directory alone will not suffice, because zfskeys is hierarchically at /etc/rc.d/, and so on.

hier(7)

rc(8): FILES

github.com

freebsd-src/libexec/rc/rc.d/zfskeys at main · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src
github.com github.com

zfs-load-key.8 — OpenZFS documentation

HTH
 
I'm wanting to do the same thing... But I want it so I can boot freebsd from linux using loader.kboot...

So are you wanting to know how to load keys so that / can be mounted from the encrypted rpool/ROOT/freebsd pool with bpool/BOOT/freebsd coming from there? You'd need at least the kernel there, which isn't a problem. The problem is getting / mounted w/o the ZFS crypto keys.

As has been pointed out, I'd be most tempted to create a small root on rpool/ROOT/freebsd that's used to load the zfs keys (not sure where your key store is) for the target root system, then reboot -r to that new partition. It's certainly doable with a custom /etc/rc script and some fiddling about, but there's nothing out of the box. You could even encode the root filesystem as a MFS into the kernel, though I don't think you could easily reclaim that memory. You might be able to reclaim one you loaded as a MFS root from rpool/ROOT/freebsd, but maybe not: kldload only deals with .ko's, not arbitrary files loaded by the loader.
 
Also, I switch between several different BEs in the boot loader... not sure the lua can't select zpool is 100% correct. I'll have to falsify it with a vm or something.
 
Quest completed! Writing you from the new FreeBSD system running off of a native encrypted ZFS root.

I ended up getting an assist from kevans in the #storage channel of the FreeBSD discord chat. Kyle correctly identified the problem being the contents of my boot pool not having a /boot directory, and also gave me the critical missing piece for my mental model - the fact that reboot -r is how I can get the running kernel to swap out the filesystem root, after loading the key for my encrypted datasets.


For those curious to do something like this themselves, I'll start to mostly refer to final encrypted root ZFS locations, for clarity, and in my final system I've got mounts like these

rpool/ROOT/freebsd /
bpool/BOOT/freebsd /boot
rpool/fbsd/... /usr /var ... and so on



Note that my bpool/BOOT/freebsd gets mounted to /boot, but it will also be the initial unencrypted root /. So I don't have just the contents of /boot in there, and I symlink an additional /boot/boot to point to . (the same directory). That way, when it initially gets mounted as the root partition of the minimal system, it has its own /boot directory within it, and at the same time, later when it gets remounted as /boot for the complete encrypted root system, its contents can be modified as usual.

To flesh out the initial minimal booting system, I used the live image to unlockand mount both bpool/BOOT/freebsd and rpool/ROOT/freebsd , and copying the contents of /etc /sbin /bin /dev /lib /libexec
from the encrypted install to /boot. And I tweaked the /boot/etc/rc.conf[ICODE] to be minimal and just have [ICODE]hostname="..." and zfs_enable="YES".

That was sufficient for me to be able to boot into single usermode on the
minimal partition, and then run
zfs load-key rpool
kenv vfs.root.mountfrom=zfs:rpool/ROOT/freebsd
reboot -r

to get into the complete encrypted system.

This was a good feeling! and at shortly thereafter back in the #storage channel,
bsdimp chimed in with:
@pi did you automate it?
Click to expand...

So here are the details of how I automated the two-phase boot.

Onto my [/ICODE]/boot/usr/bin[/ICODE], I copied some utilities from /usr/bin, specifically
cut du find grep less more sed tr vi

Though you'll see below that I only ended up using grep in the end, and that's the only one of the bunch above that needs a library beyond what in /lib, and so I also grabbed /usr/lib/libregex.*[ICODE] and stuck it into [ICODE]/boot/usr/lib

My /boot/loader.conf looks like this
cryptodev_load="YES"
zfs_load="YES"
kernel_options="-s"
boot_single="YES"
vfs.root.mountfrom="zfs:bpool/BOOT/freebsd"
init_shell="/bin/.garbage"

And the contents of /boot/bin/.garbage are

#!/bin/sh
zpool list | grep -q rpool || zpool import -f rpool
zfs load-key rpool
kenv vfs.root.mountfrom=zfs:rpool/ROOT/freebsd
reboot -r


The last thing I did was hardlink /bin/.garbage to /bin/sh.

So now, I have a system that initially boots into single user mode, prompts me for the encryption key, and proceeds to boot into the complete system.

Thanks again everyone for your help.
 
oh, and I tried to edit the prefix of the thread to mark it as solved, but perhaps because I'm a new user, or because I already changed the title of the thread once, I do not see the option to edit the thread title now.
 
Thanks!

ivanov said:
… reboot -r …
Click to expand...

Keyword, for search purposes: reroot

ivanov said:
… I do not see the option to edit the thread title now.
Click to expand...

Try:

forums.freebsd.org

Hint: how to edit the title (subject line) of a topic (thread), or add the 'Solved' prefix

A number of topics are solved, but not marked as such. There seems to be no guideline. Please, can one be added? With a screenshot, or two, to make XenForo less ambiguous. https://forums.freebsd.org/forums/forum-rules-and-guidelines-required-reading.49/ Thanks
forums.freebsd.org forums.freebsd.org
 
grahamperrin said:
Try:

forums.freebsd.org

Hint: how to edit the title (subject line) of a topic (thread), or add the 'Solved' prefix

A number of topics are solved, but not marked as such. There seems to be no guideline. Please, can one be added? With a screenshot, or two, to make XenForo less ambiguous. https://forums.freebsd.org/forums/forum-rules-and-guidelines-required-reading.49/ Thanks
forums.freebsd.org forums.freebsd.org
Click to expand...
Thanks - I came across your helpful post, but I don't see a hamburger / ellipsis menu like that, I only see a the "unwatch" option
 

Attachments

  • 2024-01-23T17:07:00-08:00.png
    2024-01-23T17:07:00-08:00.png
    5 KB · Views: 37
You must log in or register to reply here.
Back
Top