Hello Free Beastie Besties,
I have a two zfs pool root-on-ZFS Debian 12.4 setup: 1G unecrypted `bpool` and 100G zfs native encrypted `rpool .
(I got those names from following the framework in this guide: https://openzfs.github.io/openzfs-docs/Getting Started/Debian/Debian Bullseye Root on ZFS.html )
I'd like to add FreeBSD to the mix and install it into the same ZFS pools. The end goal for me is to be able to dual boot FreeBSD and Debian. It would be nice to share /home across the two operating systems, but not a dealbreaker so long as I can get access to say /debian/home/username while in FreeBSD, and vice versa, I can manage some symlinks so that the home away from home still feels appropriately cozy.
From information I've gathered it seems that this is possible, but I'm not super clear on the details. I've so far verified that I am able to boot into FreeBSD 14.0 installation media (via Ventoy with an assist from this thread https://github.com/ventoy/Ventoy/issues/2636 issuecomment -1858203099 ), and I am able to `zpool import -f` both pools (after `zfs load-key rpool`) under FreeBSD running from the installation disk.
Next I used the shell mode partitioning, created the FreeBSD filesystem datasets (eschewing the `/default` convention and using `bpool/BOOT/freebsd` for `/boot` the rest on the encrypted zpool). I ended up this these filesystem datasets:
populated `/tmp/bsdinstall_etc/fstab`, and mounted the file systems under `/mnt`, and ran the rest of the installation. But this is where things got a bit fuzzy, and I just started trying stuff. (Most other people describing encrypted root-on-ZFS under FreeBSD were using GELI, but that's not what I'm trying to do here)
I saw mention somewhere that to accomplish dual-booting Debian 12.4 / FreeBSD 14.0, that I need to use all legacy mountpoints for the OS directories ( in other words the ones in `heir(7)` ), and that I would use `/etc/fstab`
to specify what gets mounted where. If that's correct, then I'm not clear on how I can get to `/etc/fstab` if root is encrypted. It seems like I want to get to a state similar to how I boot into Debian where I boot using a dataset on the unencryted `bpool`, get a prompt for unlocking the key of the encrypted zpool, and carry on booting the system from there.
I copied the freebsd bootloader to the EFI partition, and perhaps not too unsuprisingly, it fails to boot. At least not without some manual intervention. First I end up in the EFI loader
The zio_read/ZFS errors are repeated a few times, with "Failed to find bootable partition" inserted in there, and ending with a "ERROR: cannot open /boot/lua/loader.lua: no such file or directory".
I can `set currdev="zfs:bpool/BOOT/freebsd:"`, and then have tried `load kernel/kernel` followed by `boot`, or also
sprinkling in `load kernel/zfs.ko` just in case, but end up with a system that looks like its starting to boot, but then
gives up and just reboots itself.
Sure would appreciate some guidance on how to complete this quest. Thank you!
I have a two zfs pool root-on-ZFS Debian 12.4 setup: 1G unecrypted `bpool` and 100G zfs native encrypted `rpool .
(I got those names from following the framework in this guide: https://openzfs.github.io/openzfs-docs/Getting Started/Debian/Debian Bullseye Root on ZFS.html )
I'd like to add FreeBSD to the mix and install it into the same ZFS pools. The end goal for me is to be able to dual boot FreeBSD and Debian. It would be nice to share /home across the two operating systems, but not a dealbreaker so long as I can get access to say /debian/home/username while in FreeBSD, and vice versa, I can manage some symlinks so that the home away from home still feels appropriately cozy.
From information I've gathered it seems that this is possible, but I'm not super clear on the details. I've so far verified that I am able to boot into FreeBSD 14.0 installation media (via Ventoy with an assist from this thread https://github.com/ventoy/Ventoy/issues/2636 issuecomment -1858203099 ), and I am able to `zpool import -f` both pools (after `zfs load-key rpool`) under FreeBSD running from the installation disk.
Next I used the shell mode partitioning, created the FreeBSD filesystem datasets (eschewing the `/default` convention and using `bpool/BOOT/freebsd` for `/boot` the rest on the encrypted zpool). I ended up this these filesystem datasets:
bpool/BOOT/debian
bpool/BOOT/freebsd
rpool/BOOT/
rpool/ROOT/debian
rpool/ROOT/freebsd
rpool/fbsd/tmp
rpool/fbsd/usr
rpool/fbsd/usr/home
rpool/fbsd/usr/obj
rpool/fbsd/usr/ports
rpool/fbsd/usr/ports/distfiles
rpool/fbsd/usr/ports/packages
rpool/fbsd/usr/src
rpool/fbsd/var
rpool/fbsd/var/audit
rpool/fbsd/var/crash
rpool/fbsd/var/log
rpool/fbsd/var/mail
rpool/fbsd/var/tmp
rpool/...# other debian mount points
populated `/tmp/bsdinstall_etc/fstab`, and mounted the file systems under `/mnt`, and ran the rest of the installation. But this is where things got a bit fuzzy, and I just started trying stuff. (Most other people describing encrypted root-on-ZFS under FreeBSD were using GELI, but that's not what I'm trying to do here)
I saw mention somewhere that to accomplish dual-booting Debian 12.4 / FreeBSD 14.0, that I need to use all legacy mountpoints for the OS directories ( in other words the ones in `heir(7)` ), and that I would use `/etc/fstab`
to specify what gets mounted where. If that's correct, then I'm not clear on how I can get to `/etc/fstab` if root is encrypted. It seems like I want to get to a state similar to how I boot into Debian where I boot using a dataset on the unencryted `bpool`, get a prompt for unlocking the key of the encrypted zpool, and carry on booting the system from there.
I copied the freebsd bootloader to the EFI partition, and perhaps not too unsuprisingly, it fails to boot. At least not without some manual intervention. First I end up in the EFI loader
Ignoring Boot0004: Only one DP found
Trying ESP: PciRoot(0x0).../HD(2,GPT...)
Setting curdev to disk0p2
Trying: PciRoot(0x0).../HD(3,GPT...)
Setting curdev to zfs:bpool/BOOT/freebsd:
Trying: PciRoot(0x0).../HD(4,GPT...)
Setting curdev to zfs:rpool:
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
The zio_read/ZFS errors are repeated a few times, with "Failed to find bootable partition" inserted in there, and ending with a "ERROR: cannot open /boot/lua/loader.lua: no such file or directory".
I can `set currdev="zfs:bpool/BOOT/freebsd:"`, and then have tried `load kernel/kernel` followed by `boot`, or also
sprinkling in `load kernel/zfs.ko` just in case, but end up with a system that looks like its starting to boot, but then
gives up and just reboots itself.
Sure would appreciate some guidance on how to complete this quest. Thank you!