Other "beadm" boot environments option in bootloader menu missing - Is it because geli encrypted zroot?

[omardevonlittle]

Hello everyone,

I recently reinstalled my Laptop and I am playing around on the installation. My previous install was done following this guide.

However, having to keep this USB around on my lap, having it accidentally detach sometimes or needing to FSCK it, or having to syncronize the "/boot" folder between other backups got annoying quick.

Now I just done a quick and dirty setup to see if it's possible to have dual booting on a laptop with two disks, and have FreeBSD on GELI. This consists of the following:

• Windows on nda0. It created the needed EFI partition
• An unencrypted UFS partition which holds "/boot" and everything inside it. This partition also holds the GELI key.
• Two GELI partitions across nda0 and nda1, which are mirrored to create "zroot"

The following happens when booting (as far as I understand it - I am not an expert):

• UEFI reads the "loader.efi" on "/efi/EFI/FreeBSD/loader.efi", which then searches for a boot partition. This summarizes stage 1
• It finds the unencrypted UFS partition, and loads the boot loader and reads my "/boot/loader.conf" file. This would be stage 2 (and here it would also read the boot environments)
• The bootloader now loads the kernel and sees which devices are available on the computer. This is stage 3
• Thanks to "loader.conf", the kernel asks the user for the password to decrypt the GELI partitions, and import the zroot pool and continue on with "rc". This completes stage 4.

However, I just realized that on the bootloader, I don't see the "[8] Boot Environments" option. I was wondering if I messed something up with my UFS partition, but then I realized, the zpool (and the zpool/ROOT/<beName>) are not visible yet, because they are encrypted. I just want to confirm if this is the reason why the menu is not appearing. If this is the case, then I guess that is a big disadvantage of using GELI on zroot.

I am also curious to see how you handle beadm, zroot and encryption. I would love to hear your experiences and how you setup your workstation

Finally, if someone is interested in a setup guide on what I am thinking to do with my setup, I can clean up my instructions and set it up.

 

[T-Daemon]

My previous install was done following this guide.

After reading the linked howto, a question rises: Why create a so called "extra protection" in form of a geli(8) key file (besides a second user key, in form of a passphrase), and store that key file on the same hard disk, on an un-encrypted partition, next to the encrypted partition. In this scenario, it's hardly possible to speak of extry protection when the key file is located on the same hard drive as the encrypted partition and is automatically loaded.

It's like having a safe deposit box with two locks, a keypad, and a physical keyhole, with the key already inserted in the keyhole, turned and unlocked automatically when the safe is touched, before entering the keypad combination.

A bad actor can eventually destroy the key file, and the owner of the encrypted provider has to restore it from a backup.

If you want that extra protection key file, then you put it on a removable media, not on the same device, besides the encrypted partition, on an un-encrypted file system.

In the above setup you could just as well use a passphrase only encryption.

Ditch the key file and the un-encrypted /boot, use a passphrase only full disk (partition) encrypted Root-on-ZFS (including /boot). This would resolve the boot environment issue.

When initializing the geli(8) provider, use the "-g" option.

I am also curious to see how you handle beadm, zroot and encryption. I would love to hear your experiences and how you setup your workstation

I did setup a passphrase only geli(8) encrypted Root-on-ZFS installation on a laptop, besides a Linux installation on the same disk. Both systems are started via the machines UEFI disc boot menu.

Boot environments are handled by bectl(8) mostly, having sysutils/beadm installed, but don't use it often.

 

[omardevonlittle]

Why create a so called "extra protection" in form of a geli(8) key file (besides a second user key, in form of a passphrase), and store that key file on the same hard disk, on an un-encrypted partition, next to the encrypted partition

That's true. In this scenario, the key file serves no purpose besides adding complexity. Only in another scenario can the key be useful, but then again, it adds further complexity that my workstation laptop probably doesn't need. As I've read on the MWL books, adding more complex layers on encryption will not make it stronger, will make the user dizzier and more likely to lose his data.

Ditch the key file and the un-encrypted /boot, use a passphrase only full disk (partition) encrypted Root-on-ZFS (including /boot). This would resolve the boot environment issue.

I recall now when I was first trying this out on a VM that FreeBSD asks for the GELI passphrase on stage 2, before the boot loader loads, and from here read the ZFS datasets and boot environments.

Thanks for the suggestion, I will read some more online. If you also have a guide that you use when installing it on shell it would help me a lot

EDIT: Just for curiosity sake, say I wanted to separate the GELI key on an USB device, and only have the key on the device (no boot partition), how would I provide it to FreeBSD so that it can mount the USB device and read the key? Would this be done on stage 2 or stage 3? In this case maybe an unencrypted /boot is needed so we can add more loader.conf statements and modify the GELI prompt, right?

 

[T-Daemon]

If you also have a guide that you use when installing it on shell it would help me a lot

Sure, see following guide I wrote. This gives me the opportunity to practice and memorize how to do it when I need to perform such a procedure myself.

Boot up the FreeBSD installer media, start the menu guided installation, configure keymap, hostname, components to install, at the "Partioning" menu enter "Shell" and begin the system installation preparations, as they are:

• modify partition table (gpart(8)), add freebsd-swap, freebsd-zfs partitions
• add FreeBSD loader to a existing UEFI ESP
• create a FreeBSD UEFI menu entry
• initialize/attach geli(8) provider
• zpool-create(8)) pool. Make sure the pool is mounted at /mnt for system installation (note the comment after dropping to shell), and configuration by bsdinstall(8).)
• create ZFS datasets (here you can create the datasets manually or execute a pre-configured script. See example down below).
• edit /tmp/bsdinstall_boot/loader.conf
• edit /tmp/bsdinstall_etc/fstab
• edit /tmp/bsdinstall_etc/rc.conf.zfs
• edit /tmp/bsdinstall_etc/sysctl.conf
• exit "Shell". By exiting the shell, the menu guided installation resumes, the system files are installed, system configuration scripts executed (network, timezone, user, etc.)

After the installation has finished, at the "Manual Configuration" menu, open a shell (enter "YES") and check /boot/loader.conf if all important kernel modules have been set, geom_eli and zfs in particular, check /etc/rc.conf, /etc/sysctl.conf. Check also for double entries.

Example of installation preparation steps

# modify partition table
gpart add -t freebsd-swap -a 1m -s 4g -l swap0 nda0
gpart add -t freebsd-zfs -a 1m -l zfs0 nda0

# add FreeBSD loader
mount_msdos  /dev/nda0p1  /mnt
mkdir -p /mnt/efi/freebsd
cp /boot/loader.efi  /mnt/efi/freebsd
umount /mnt

# create UEFI menu entry
efibootmgr -c -a -L FreeBSD -l nda0p1:/efi/freebsd/loader.efi

# initialize, attach geli(8) provider
geli init -g -l 256 -s 4096 nda0p5
geli attach nda0p5

# create ZFS pool
zpool create -o ashift=12 -o altroot=/mnt -O compress=lz4 -O atime=off -m none zroot nda0p5.eli

# create manually or scripted datasets (see example script down below).
# For scripted, or acquire IP lease, fetch script from remote machine
# and execute, or execute from plugged in, mounted USB stick.

dhclient em0
scp <user>@1.2.3.4:/tmp/zfs-dataset-creator.sh /tmp/zfs-d-c.sh

sh /tmp/zfs-d-c.sh

# configure system bootstrap required kernel modules

ee /tmp/bsdinstall_boot/loader.conf
  kern.geom.label.disk_ident.enable="0"
  kern.geom.label.gptid.enable="0"
  geom_eli_load="YES"
  # the following modules should not need to be set; they are set automatically
  # by bsdinstall(8) scripts.
  cryptodev_load="YES"
  zfs_load="YES"

# configure fstab, configure swap to be encrypted

ee /tmp/bsdinstall_etc/fstab
  /dev/nda0p1       /boot/efi   msdosfs  rw    2   2
  /dev/nda0p4.eli   none        swap     sw    0   0
# instead of device names, device labels can be used instead (i.e. /dev/gpt/efi0, /dev/gpt/swap0)

ee /tmp/bsdinstall_etc/rc.conf.zfs
  zfs_enable="YES"

ee /tmp/bsdinstall_etc/sysctl.conf
  vfs.zfs.min_auto_ashift=12

# exit shell to resume automatic system installation and menu guided configuration
exit

Example ZFS dataset creation and configuration script (extracted from a installation log /var/log/bsdinstall_log). See also the ZFS installation and configuration script for default datasets (/usr/libexec/bsdinstall/zfsboot):

#!/bin/sh

# Creating ZFS datasets
zfs create -o mountpoint=none zroot/ROOT

zfs create -o mountpoint=/ zroot/ROOT/default

zfs create -o mountpoint=/home zroot/home

zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp

zfs create -o mountpoint=/usr -o canmount=off zroot/usr

zfs create -o setuid=off zroot/usr/ports

zfs create  zroot/usr/src

zfs create -o mountpoint=/var -o canmount=off zroot/var

zfs create -o exec=off -o setuid=off zroot/var/audit

zfs create -o exec=off -o setuid=off zroot/var/crash

zfs create -o exec=off -o setuid=off zroot/var/log

zfs create -o atime=on zroot/var/mail

zfs create -o setuid=off zroot/var/tmp

# Setting mountpoint for root of the pool
zfs set mountpoint=/zroot zroot

# Modifying directory permissions
mkdir -p /mnt/tmp

chmod 1777 /mnt/tmp

mkdir -p /mnt/var/tmp

chmod 1777 /mnt/var/tmp

# Setting bootfs property
zpool set bootfs=zroot/ROOT/default zroot

# Configuring zpool.cache for zroot
mkdir -p /mnt/boot/zfs

zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot

# Set canmount=noauto for any datasets under the BE
zfs set canmount=noauto zroot/ROOT/default

Modify script if datasets are missing, or to many, or to change properties.

Before applying the installation as described on real hardware, practice in a VM, if possible, to minimize failure.

EDIT: Just for curiosity sake, say I wanted to separate the GELI key on an USB device, and only have the key on the device (no boot partition), how would I provide it to FreeBSD so that it can mount the USB device and read the key? Would this be done on stage 2 or stage 3? In this case maybe an unencrypted /boot is needed so we can add more loader.conf statements and modify the GELI prompt, right?

Right. The FreeBSD loader doesn't support mounting file system before the kernel is loaded, an unencrypted /boot might be one solution.

Another option is to separate the operation system and the important database on two encrypted partitions. The OS partition has a passphrase user key only, the database a variety of (multiple) key file locations, and user key combinations (passphrase and key file combined or individually separated, see geli(8) "setkey -n" option).

After the OS has booted in the above scenario, key files from external device(s) can be inserted to automatically (or in combination passphrase, key file) attach the encrypted partition and import the pool(s), for example, via fstab(5) and /etc/rc.conf (see /etc/defaults/rc.conf "# GELI disk encryption configuration."), or devd.conf(5)

 

[omardevonlittle]

Sure, see following guide I wrote. This gives me the opportunity to practice and memorize how to do it when I need to perform such a procedure myself.

Thank you very much for this! It will be very useful for when I start experimenting again

So far on my personal laptop I've reinstalled FreeBSD as you have recommended me, a passphrase only GELI partition where the OS is all inside this partition. Now when I installed "beadm" and I made a BE, the bootloader menu to change boot environments appears!