ZFS GPT labels for GELI encrypted root

Hi,

Just installed a new FreeBSD server. GELI encrypted ZFS-on-root. FreeBSD 15.0-RELEASE.

loader.conf

Code:
geom_mirror_load="YES"
geom_eli_load="YES"
geli_gpt_beastie-sysroot0_keyfile0_load="YES"
geli_gpt_beastie-sysroot0_keyfile0_type="gpt/beastie-sysroot0:geli_keyfile0"
geli_gpt_beastie-sysroot0_keyfile0_name="/boot/keys/beastie-sysroot_encryption.key"
geli_gpt_beastie-sysroot1_keyfile0_load="YES"
geli_gpt_beastie-sysroot1_keyfile0_type="gpt/beastie-sysroot1:geli_keyfile0"
geli_gpt_beastie-sysroot1_keyfile0_name="/boot/keys/beastie-sysroot_encryption.key"


Upon boot I get asked to enter passphrase for a lot of partitions -- how can I make this stop so that it only asks about the partitions that I have specified in loader.conf?


Here is an example how it might look:
Code:
EOM_ELI: Wrong key for ada1p2. Tries left: 2.
Enter passphrase for ada1p2: GEOM_ELI: Wrong key for ada1p2. Tries left: 1.
Enter passphrase for ada1p2: GEOM_ELI: Wrong key for ada1p2. No tries left.
GEOM_MIRROR: Device mirror/swap launched (1/2).
Enter passphrase for ada1p3: GEOM_ELI: Wrong key for ada1p3. Tries left: 2.
Enter passphrase for ada1p3: GEOM_ELI: Wrong key for ada1p3. Tries left: 1.
Enter passphrase for ada1p3: GEOM_ELI: Wrong key for ada1p3. No tries left.
Enter passphrase for ada3p2: GEOM_ELI: Wrong key for ada3p2. Tries left: 2.
Enter passphrase for ada3p2: GEOM_ELI: Wrong key for ada3p2. Tries left: 1.
Enter passphrase for ada3p2: GEOM_ELI: Wrong key for ada3p2. No tries left.
Enter passphrase for ada3p3: GEOM_ELI: Wrong key for ada3p3. Tries left: 2.
Enter passphrase for ada3p3: GEOM_ELI: Wrong key for ada3p3. Tries left: 1.
Enter passphrase for ada3p3: GEOM_ELI: Wrong key for ada3p3. No tries left.
Enter passphrase for gpt/beastie-sysroot0: 
GEOM_ELI: Device gpt/beastie-sysroot0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: accelerated software
Enter passphrase for gpt/beastie-storage0: 
GEOM_ELI: Wrong key for gpt/beastie-storage0. Tries left: 2.
Enter passphrase for gpt/beastie-storage0: 
GEOM_ELI: Wrong key for gpt/beastie-storage0. Tries left: 1.
Enter passphrase for gpt/beastie-storage0: 
GEOM_ELI: Wrong key for gpt/beastie-storage0. No tries left.
Enter passphrase for gptid/006a0905-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/006a0905-5f70-11f0-9df8-941882376644. Tries left: 2.
Enter passphrase for gptid/006a0905-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/006a0905-5f70-11f0-9df8-941882376644. Tries left: 1.
Enter passphrase for gptid/006a0905-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/006a0905-5f70-11f0-9df8-941882376644. No tries left.
Enter passphrase for gpt/beastie-sysroot1: 
GEOM_ELI: Device gpt/beastie-sysroot1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: accelerated software
Enter passphrase for gpt/beastie-storage1: GEOM_ELI: Wrong key for gpt/beastie-storage1. Tries left: 2.
Enter passphrase for gpt/beastie-storage1: GEOM_ELI: Wrong key for gpt/beastie-storage1. Tries left: 1.
Enter passphrase for gpt/beastie-storage1: GEOM_ELI: Wrong key for gpt/beastie-storage1. No tries left.
Enter passphrase for gptid/2d1a05c3-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/2d1a05c3-5f70-11f0-9df8-941882376644. Tries left: 2.
Enter passphrase for gptid/2d1a05c3-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/2d1a05c3-5f70-11f0-9df8-941882376644. Tries left: 1.
Enter passphrase for gptid/2d1a05c3-5f70-11f0-9df8-941882376644: 
GEOM_ELI: Wrong key for gptid/2d1a05c3-5f70-11f0-9df8-941882376644. No tries left.
GEOM_ELI: Device mirror/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: accelerated software
 
Thanks for your reply!

My problem is that ada1p2 is the same as gpt/beastie-sysroot0 -- what happens if I then remove the BOOT-flag? (There are no assigned GELIBOOT-flags at the moment)

Will it still ask me once during the boot process -- but for the GPT-label?
 
if one of the flags is present and the cached password wont work it will ask.
thats what i understood from the kernel source
if there are no flags the provider will be skipped and such not usable in fstab
 
If you tell us about the current geli(8) Root-on-ZFS, swap disk setup, and how the geli providers were initialized (passphrase, key files, in combination, key files only, etc.), it would be much easier than guessing the setup and configuration.

My problem is that ada1p2 is the same as gpt/beastie-sysroot0 -- what happens if I then remove the BOOT-flag? (There are no assigned GELIBOOT-flags at the moment)

If the current Root-on-ZFS is not configured to boot from a encrypted root file system (GELIBOOT), how is the system booted then?

I see there are gmirror(8)'ed, geli(8) encrypted swap. Tell us about them as well. Why gmirror swap in the first place?

Show us the storage configuration:

gpart info -lp

After boot:

geli status

geli list


If you could explain to us what kind of disk security (e.g. OS, data storage disk encryption, etc.) you have in mind, we can make suggestions. Is the machine to be used a desktop PC (monitor, keyboard attached), or a further away headless server (e.g. basement, off location)?
 
I am booting from a small SSD disks that has an unencrypted bootpool on zfs:
Code:
# gpart show ada4
=>       40  250069600  ada4  GPT  (119G)
         40     532480     1  efi  (260M)
     532520       1024     3  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528    2097152     2  freebsd-zfs  (1.0G)
    2631680  247437960        - free -  (118G)
(system is BIOS booting and not UEFI, but that should be unrelated to my current issue)

Then I have encrypted Root-on-ZFS -- "gpart info" wasn't available so I am pasting standard show here:
Code:
# gpart show
=>        40  7814037088  ada1  GPT  (3.6T)
          40        2008        - free -  (1.0M)
        2048    33554432     1  freebsd-swap  (16G)
    33556480   629145600     2  freebsd-zfs  (300G)
   662702080  7109345280     3  freebsd-zfs  (3.3T)
  7772047360    41989768        - free -  (20G)

=>       40  250069600  ada4  GPT  (119G)
         40     532480     1  efi  (260M)
     532520       1024     3  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528    2097152     2  freebsd-zfs  (1.0G)
    2631680  247437960        - free -  (118G)

=>        40  7814037088  ada3  GPT  (3.6T)
          40        2008        - free -  (1.0M)
        2048    33554432     1  freebsd-swap  (16G)
    33556480   629145600     2  freebsd-zfs  (300G)
   662702080  7109345280     3  freebsd-zfs  (3.3T)
  7772047360    41989768        - free -  (20G)

Code:
# geli status
                     Name  Status  Components
gpt/beastie-sysroot0.eli  ACTIVE  gpt/beastie-sysroot0
gpt/beastie-sysroot1.eli  ACTIVE  gpt/beastie-sysroot1
          mirror/swap.eli  ACTIVE  mirror/swap
          
          
# geli list
Geom name: gpt/beastie-sysroot0.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, AUTORESIZE
KeysAllocated: 75
KeysTotal: 75
Providers:
1. Name: gpt/beastie-sysroot0.eli
   Mediasize: 322122543104 (300G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: gpt/beastie-sysroot0
   Mediasize: 322122547200 (300G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17180917760
   Mode: r1w1e1

Geom name: gpt/beastie-sysroot1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, AUTORESIZE
KeysAllocated: 75
KeysTotal: 75
Providers:
1. Name: gpt/beastie-sysroot1.eli
   Mediasize: 322122543104 (300G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: gpt/beastie-sysroot1
   Mediasize: 322122547200 (300G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17180917760
   Mode: r1w1e1

Geom name: mirror/swap.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 4
KeysTotal: 4
Providers:
1. Name: mirror/swap.eli
   Mediasize: 17179865088 (16G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: mirror/swap
   Mediasize: 17179868672 (16G)
   Sectorsize: 512
   Mode: r1w1e1

Both these are encrypted with both keyfile and keyphrase:
  • gpt/beastie-sysroot0
  • gpt/beastie-sysroot1

Swap is mirrored to achieve theoretically higher read speeds. This is a headless server where I enter password over BMC. I run similar setups on other systems -- what's new here is that it asks me for password for devices that I have not specified in loader.conf.


To clarify, when I boot this happens:
  • System asks 3 times for password for: ada1p2 (same as gpt/beastie-sysroot0) -- I don't want system to ask for password for this during boot.
  • System asks 3 times for password for: ada1p3 (same as gpt/beastie-sysroot1) -- I don't want system to ask for password for this during boot.
  • System asks 3 times for password for: ada3p2 -- I don't want system to ask for password for this during boot.
  • System asks 3 times for password for: ada3p3 -- I don't want system to ask for password for this during boot.
  • System asks for password for: gpt/beastie-sysroot0 -- entering correct password decrypts it and attaches it to zpool.
  • System asks for password for: gpt/beastie-sysroot1 -- entering correct password decrypts it and attaches it to zpool.
 
"gpart info" wasn't available so I am pasting standard show here:
My mistake, muscle memory from pkg info, what I meant was gpart show -lp

  • System asks 3 times for password for: ada1p2 (same as gpt/beastie-sysroot0) -- I don't want system to ask for password for this during boot.
  • System asks 3 times for password for: ada1p3 (same as gpt/beastie-sysroot1) -- I don't want system to ask for password for this during boot.
I believe GPT device nodes are not created in the early stage of the boot process (notice how geli is asking for the ada(4) device names instead of the GPT labeled device names in your opening post), because of this it fails to decrypt the device, despite the right passphrase, because the loader.conf configuration is done with GPT labels, but later in the boot process it succeeds with the GPT labels, after the GPT device nodes are created.

You need to set ada(4) device names in /boot/loader.conf
Rich (BB code):
- geli_gpt_beastie-sysroot0_keyfile0_type="gpt/beastie-sysroot0:geli_keyfile0"
+ geli_gpt_beastie-sysroot0_keyfile0_type="ada1p2:geli_keyfile0"

- geli_gpt_beastie-sysroot0_keyfile1_type="gpt/beastie-sysroot1:geli_keyfile0"
+ geli_gpt_beastie-sysroot0_keyfile0_type="ada1p3:geli_keyfile0"

  • System asks 3 times for password for: ada3p2 -- I don't want system to ask for password for this during boot.
  • System asks 3 times for password for: ada3p3 -- I don't want system to ask for password for this during boot.
I assume these are gpt/beastie-storage0 and gpt/beastie-storage1, as long as the BOOT flag is set it will ask for a passphrase during boot. If they are not automatically attached during boot, just remove the BOOT flag (geli(8) configure -B <provider>).


If you insist on GPT labels for "beastie-sysroot", configure in /etc/rc.conf:
Code:
# GELI disk encryption configuration.
geli_devices=""         # List of devices to automatically attach in addition to
                        # GELI devices listed in /etc/fstab.
geli_groups=""          # List of groups containing devices to automatically
                        # attach with the same keyfiles and passphrase
geli_tries=""           # Number of times to attempt attaching geli device.
                        # If empty, kern.geom.eli.tries will be used.
geli_default_flags=""   # Default flags for geli(8).
geli_autodetach="YES"   # Automatically detach on last close.
                        # Providers are marked as such when all file systems are
                        # mounted.
# Example use.
#geli_devices="da1 mirror/home"
#geli_da1_flags="-p -k /etc/geli/da1.keys"
#geli_da1_autodetach="NO"
#geli_mirror_home_flags="-k /etc/geli/home.keys"
#geli_groups="storage backup"
#geli_storage_flags="-k /etc/geli/storage.keys"
#geli_storage_devices="ada0 ada1"
#geli_backup_flags="-j /etc/geli/backup.passfile -k /etc/geli/backup.keys"
#geli_backup_devices="ada2 ada3"
 
Thanks for your reply!

But regarding the proposed changes to loader.conf -- what would be a good strategy to have a setup that doesn't require reconfiguration if there is an enumeration change of the devices? Ie: ada3 being enumerated ada2 instead.
 
Make the configuration from /etc/rc.conf using unique GPT labels as shown in the spoiler of post # 7 with a mirror label. rc.conf is executed after the GPT device nodes are created.
 
But if I added these lines to rc.conf -- would that help with decrypting root? Surely not as rc.conf would be stored in /etc on root? Or am I missing something here?
 
Code:
diff --git a/sys/geom/eli/g_eli.c b/eli/g_eli.c
index bed0875..f961301 100644
--- a/sys/geom/eli/g_eli.c
+++ b/eli/g_eli.c
@@ -1630,6 +1630,12 @@ g_eli_fini(struct g_class *mp)
                EVENTHANDLER_DEREGISTER(shutdown_pre_sync, g_eli_pre_sync);
 }

-DECLARE_GEOM_CLASS(g_eli_class, g_eli);
+#define DECLARE_GEOM_CLASS_ORD(class, name, ord)                     \
+        static moduledata_t name##_mod = {                      \
+                #name, g_modevent, &class                       \
+        };                                                      \
+        DECLARE_MODULE(name, name##_mod, SI_SUB_DRIVERS, SI_ORDER_SECOND + ord);
+
+DECLARE_GEOM_CLASS_ORD(g_eli_class, g_eli, 1);
 MODULE_DEPEND(g_eli, crypto, 1, 1, 1);
 MODULE_VERSION(geom_eli, 0);
out of curiosity can you try this patch
in theory will make geom_label run before geom_eli
 
I happily test, is this the correct way to apply it?
Code:
# cd /usr/src
/usr/src # patch -p1 < /tmp/patch.diff
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/sys/geom/eli/g_eli.c b/eli/g_eli.c
|index bed0875..f961301 100644
|--- a/sys/geom/eli/g_eli.c
|+++ b/eli/g_eli.c
--------------------------
Patching file sys/geom/eli/g_eli.c using Plan A...
Hunk #1 succeeded at 1629 with fuzz 1 (offset -1 lines).
done
 
But if I added these lines to rc.conf -- would that help with decrypting root? Surely not as rc.conf would be stored in /etc on root? Or am I missing something here?
If you mean encrypted root rc.conf, than no. This must be configured in the unencrypted root rc.conf.

And I forgot to mention, the BOOT flag must be removed when configured from rc.conf.

But I wonder, why use an unencrypted system at all? Why not encrypt gpt/beastie-sysroot0 gpt/beastie-sysroot1 GELIBOOT with a passphrase only. The keyfile doesn't add an extra security layer when it's loaded automatically from the same provider.
 
No, from the the unencrypted bootpool on ada4p2 /etc/rc.conf
I am booting from a small SSD disks that has an unencrypted bootpool on zfs:
Code:
# gpart show ada4
=>       40  250069600  ada4  GPT  (119G)
         40     532480     1  efi  (260M)
     532520       1024     3  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528    2097152     2  freebsd-zfs  (1.0G)
    2631680  247437960        - free -  (118G)
The system would ignore /boot/etc/rc.conf, see hier(7) for file system hierarchy.
 
I am booting from a small SSD disks that has an unencrypted bootpool on zfs:
So /boot/etc/rc.conf?
I just realized, do you mean by "bootpool on zfs" there is only "/boot , /boot/kernel" on it, not a full root system? Then it's not possible to simply add /etc/rc.conf on that file system.

But, again, why bother with an unencrypted /boot when the system can be booted from a geli(8) encrypted Root-on-ZFS directly. The FreeBSD loader is capable of doing that.
 
Code:
diff --git a/sys/geom/eli/g_eli.c b/eli/g_eli.c
index bed0875..f961301 100644
--- a/sys/geom/eli/g_eli.c
+++ b/eli/g_eli.c
@@ -1630,6 +1630,12 @@ g_eli_fini(struct g_class *mp)
                EVENTHANDLER_DEREGISTER(shutdown_pre_sync, g_eli_pre_sync);
 }

-DECLARE_GEOM_CLASS(g_eli_class, g_eli);
+#define DECLARE_GEOM_CLASS_ORD(class, name, ord)                     \
+        static moduledata_t name##_mod = {                      \
+                #name, g_modevent, &class                       \
+        };                                                      \
+        DECLARE_MODULE(name, name##_mod, SI_SUB_DRIVERS, SI_ORDER_SECOND + ord);
+
+DECLARE_GEOM_CLASS_ORD(g_eli_class, g_eli, 1);
 MODULE_DEPEND(g_eli, crypto, 1, 1, 1);
 MODULE_VERSION(geom_eli, 0);
out of curiosity can you try this patch
in theory will make geom_label run before geom_eli
No I have compiled kernel (first time compiling FreeBSD kernel):
Code:
cd /usr/src
# make buildkernel
# make installkernel

Result:
  • It still starts with asking for passphrase for all the ada-devices
  • However: when asking for sysroot0 it also automatically unlocks sysroot1 -- this was not the case before.
 
I just realized, do you mean by "bootpool on zfs" there is only "/boot , /boot/kernel" on it, not a full root system? Then it's not possible to simply add /etc/rc.conf on that file system.

But, again, why bother with an unencrypted /boot when the system can be booted from a geli(8) encrypted Root-on-ZFS directly. The FreeBSD loader is capable of doing that.
I run both bootpool-on-zfs as well as GELI encrypted root-on-ZFS. The reason I am doing it like this is:
  • I have been using the same install notes since I installed my first FreeBSD encrypted server in 2016 -- at that point no other way were possible (I believe) -- now adter having read this thread I realize that nowadays it is possible for the loader to unlock a geli encrypted root.
  • But: I like that there are two factors -- the physical SSD I can remove after boot as well as something to remember -- could this be accomplished if I skip a separate non-encrypted bootpool?
 
But: I like that there are two factors -- the physical SSD I can remove after boot as well as something to remember -- could this be accomplished if I skip a separate non-encrypted bootpool?
I'm not sure if I understand you correctly, but if you mean whether a geli(8) encrypted Root-on-ZFS SSD can be removed from one machine and connected to any SATA port in another machine, then this is possible.

The FreeBSD bootloader has logic to recognize any SATA port name changes, a geli(8) encrypted Root-on-ZFS provider won't refuse to boot when the device names change.

In any case, no unencrypted boot partition (full OS install, or only /boot , kernel and helper scripts) is necessary.

If you install menu-guided, encrypted Root-on-ZFS is configured automatically if you choose the "Encrypt Disks" option. If you prefer a manual installation, make sure the geli(8) -g option is set when the provider is initialized, e.g.:
Rich (BB code):
# geli init -g -l 256 -s 4096  <provider0>  <provider1>

Code:
     init       Initialize providers which need to be encrypted.  If multiple
                providers are listed as arguments, they will all be
                initialized with the same passphrase and/or User Key. ...

                ...

                -g                Enable booting from this encrypted root
                                  filesystem.  The boot loader prompts for the
                                  passphrase and loads loader(8) from the
                                  encrypted partition.


The menu-guided installation tends to use most of the disk for Root-on-ZFS "freebsd-zfs" partition. You can circumvent this restriction, by creating a huge swap partition, then, after the installation has completed, delete swap, create a smaller swap and a second "freebsd-zfs" partition in size of your choise.

For example, taking your disk setup ada1 and ada3 partition table, which seems a mirror, in the "ZFS Configuration" menu, chose a swap size of 3300G, reserve 300G for Root-on-ZFS, if you wish mirrored Root-on-ZFS, choose "Pool Type/Disks" -> "mirror", chose "Mirror Swap", "Encrypt Swap".

After the installation has finished, delete swap, add a 16G swap on each disk, create geli(8) encrypted storage0 and storage1 "freebsd-zfs" partitions, zpool-create(8) them. Make sure the encrypted swap devices are properly configured in /etc/fstab, use eventually GPT labels to name them uniquely.

If you wish to decrypt storage0/1 together with the Root-on-ZFS before the kernel is booted, initialize them with the -g option and the same passphrase as Root-on-ZFS. If the setup is to decrypt them after Root-on-ZFS has fully booted into multi-user mode, don't set -g -b options.
 
The FreeBSD bootloader has logic to recognize any SATA port name changes, a geli(8) encrypted Root-on-ZFS provider won't refuse to boot when the device names change.
But if this is the case -- why in the boot process is not the passphrase accepted for ada1p2 (same as gpt/beastie-sysroot0) -- but it is accepted when it is asked for gpt/beastie-sysroot0? My understanding would be that it is because I have defined the keyfile for gpt/beastie-sysroot0 but not for ada1p2 in loader.conf.
 
why in the boot process is not the passphrase accepted for ada1p2 (same as gpt/beastie-sysroot0) -- but it is accepted when it is asked for gpt/beastie-sysroot0?
The geli(8) kernel module cannot handle the GPT labeled device names used for the configuration in loader.conf but only by the device driver created device node names ( /dev/ada* ATA device nodes by ada(4) , same with nda(4) , da(4)) in the early stage of the boot process, when the GPT device labeled device nodes are not created yet, but later in the boot process when GPT device labels nodes are created, it would accept the passphrase.

If I were you, I would stop dwelling on this problem and instead use the direct encrypted Root-on-ZFS method. I have such a configuration right now on a laptop. It's no different on a server or desktop machine.
 
Back
Top