ZFS Changes to /etc/rc.d/zfskeys stops 15.0 system from booting

subnetspider

subnetspider

Today I upgraded one of my FreeBSD NAS servers from 14.4-RELEASE to 15.0-RELEASE, which worked fine, but after upgrading the userland and rebooting, the system never came back online.

After connecting a keyboard and monitor, FreeBSD now seems to prompt you to enter the encryption key of encrypted datasets with the ZFS property keylocation:promt, instead of skipping them and displaying an error message.

This results in the boot process halting indefinitely, until you a) manually enter the passphrase, or b) manually press Ctrl + C, which makes the system unaccessible via SSH.

The changes responsible for this behavior are located in the function unlock_fs() in the script /etc/rc.d/zfskeys and are not mentioned in the 15.0 release notes at all.

Function in 14.4-RELEASE:

Code: 
unlock_fs()
{
    local fs="$1"
    local kl="$2"
    local k="${kl##file://}"
    if [ "$k" ] && [ -f "$k" ] && [ -s "$k" ] && [ -r "$k" ]; then
        if [ "$(zfs get -Ho value keystatus "$fs")" = 'available' ]; then
            echo "Key already loaded for $fs."
        elif keytest=$(zfs load-key -n -L "$kl" "$fs" 2>&1); then
            echo "Loading key for $fs from $kl.."
            if ! keyload=$(timeout $zfskeys_timeout zfs load-key -L "$kl" "$fs"
2>&1) ; then
                if [ $? -eq 124 ]; then
                    echo "Timed out loading key from $kl for $fs"
                else
                    echo "Failed to load key from $kl for $fs:"
                    echo "$keyload"
                fi
            fi
        else
            echo "Could not verify key from $kl for $fs:"
            echo "$keytest"
        fi
    else
        echo "Key file $k not found, empty or unreadable. Skipping $fs.."
    fi
}

Function in 15.0-RELEASE:

Code: 
unlock_fs()
{
    local fs="$1"
    local kl="$2"
    local k="${kl##file://}"
    if [ "$kl" == "prompt" ]
    then
        echo "Key prompt for $fs."
       if zfs load-key -L "$kl" "$fs" < /dev/tty > /dev/tty 2>/dev/tty ; then
           echo "Key loaded for $fs."
       else
           echo "Key failed to load for $fs."
        fi
    elif [ "$k" ] && [ -f "$k" ] && [ -s "$k" ] && [ -r "$k" ]; then
        if [ "$(zfs get -Ho value keystatus "$fs")" = 'available' ]; then
            echo "Key already loaded for $fs."
        elif keytest=$(zfs load-key -n -L "$kl" "$fs" 2>&1); then
            echo "Loading key for $fs from $kl.."
            if ! keyload=$(timeout $zfskeys_timeout zfs load-key -L "$kl" "$fs"
2>&1) ; then
                if [ $? -eq 124 ]; then
                    echo "Timed out loading key from $kl for $fs"
                else
                    echo "Failed to load key from $kl for $fs:"
                    echo "$keyload"
                fi
            fi
        else
            echo "Could not verify key from $kl for $fs:"
            echo "$keytest"
        fi
    else
        echo "Key file $k not found, empty or unreadable. Skipping $fs.."
    fi
}

Is there a way to make FreeBSD ignore encrypted datasets at boot again, like in previous releases, instead of halting boot indefinitely?

Alternatively, it would be nice of there was a timeout of some sort, but I guess I have to write a PR for that?
 
I can't see much discussion about the why the change was made:

⚙ D36081 zfskeys: if location is prompt then prompt during boot

reviews.freebsd.org reviews.freebsd.org

I've got 14.4 machines that boot off unencrypted drives, then I want to SSH in and type the passphrase when I'm ready to do so. That means that I do not have to have BMC access, the OS will boot and I can finish bringing the machine back online via SSH.

Sounds like this change will prevent that, so I'll have to hold off upgrades to 15.0 until I've checked.
 
I'm learning about ZFS, so there may be newbie stuff in here, and I might be dragging red herrings across your path, so take with a liberal pinch of salt. Too many cliches!

I'm not seeing this on a test FreeBSD 15.0 machine; this is not an 14.4->15.0 upgrade (I don't think, can't remember).

So I must be doing something a bit different (maybe the boot drive being UFS?). Posting this in case it helps at all.

/etc/rc.conf: zfs_enable="YES"
/boot/loader.conf: vfs.zfs.arc_max="16G"

I have a boot drive, UFS. There are no prompts for ZFS keys when the machine reboots.

After the OS has booted I SSH in, and run a script consisting of this:
Code: 
zfs load-key mydrive/database
zfs mount mydrive/database
I get prompted for the passphrase by the load-key command.

Immediately after the boot (and before running the script above), I see this:
Code: 
# zfs get all /mydrive/database/
...
mydrive  encryption              off                     default
mydrive  keylocation             none                    default
mydrive  keyformat               none                    default
...
That threw me a bit, because I was sure I had enabled encryption on this dataset, and why is the first column just "mydrive"?

After running the script, and typing the passphrase, things look as expected:
Code: 
# zfs get all /mydrive/database/
...
mydrive/database  encryption              aes-256-gcm             -
mydrive/database  keylocation             prompt                  local
mydrive/database  keyformat               passphrase              -
...
 
subnetspider said:
[…] Alternatively, it would be nice of there was a timeout of some sort, […]
Click to expand...
timeout(1) is already used, but just for the non‑prompt keys, which makes sense for http://… key locations. I think a reasonable implementation would just skip data sets that are not canmountable (zfsprops(7)).​
Diff: 
--- zfskeys~	2026-05-15 10:40:35.809988607 +0000
+++ zfskeys	2026-05-15 10:40:35.809988607 +0000
@@ -30,12 +30,12 @@
 {
     if [ "$zfskeys_args" ]; then
         echo "$zfskeys_args" | b64decode -r |
-            xargs -0 zfs get -H -s local -o value,name keylocation
+            xargs -0 zfs get -H -s local -o value,name keylocation,canmount
     elif [ ! "$zfskeys_datasets" ]; then
-        zfs get -H -t filesystem,volume -s local -o value,name keylocation
+        zfs get -H -t filesystem,volume -s local -o value,name keylocation,canmount
     else
         echo "$zfskeys_datasets" | xargs -n 1 zfs get -H -s local \
-            -o value,name keylocation
+            -o value,name keylocation,canmount
     fi
 }
 
@@ -43,15 +43,18 @@
 {
     local fs="$1"
     local kl="$2"
+    local cm="$3"
     local k="${kl##file://}"
 
+    [ "$cm" == "on" ] || return
+    
     if [ "$kl" == "prompt" ]
     then
         echo "Key prompt for $fs."
         if zfs load-key -L "$kl" "$fs" < /dev/tty > /dev/tty 2>/dev/tty ; then
-	    echo "Key loaded for $fs."
+            echo "Key loaded for $fs."
         else
-	    echo "Key failed to load for $fs."
+            echo "Key failed to load for $fs."
         fi
     elif [ "$k" ] && [ -f "$k" ] && [ -s "$k" ] && [ -r "$k" ]; then
         if [ "$(zfs get -Ho value keystatus "$fs")" = 'available' ]; then
@@ -99,7 +102,7 @@
 {
     local IFS=$(printf "\t")
 
-    list_datasets | while read kl fs ; do
+    list_datasets | while read kl fs ; read cm fs ; do
         echo "$fs: $(zfs get -Ho value keystatus "$fs")"
     done
 }
@@ -108,8 +111,8 @@
 {
     local IFS=$(printf "\t")
 
-    list_datasets | while read kl fs ; do
-        unlock_fs "$fs" "$kl"
+    list_datasets | while read kl fs ; read cm fs ; do
+        unlock_fs "$fs" "$kl" "$cm"
     done
 }
 
@@ -117,7 +120,7 @@
 {
     local IFS=$(printf "\t")
 
-    list_datasets | while read kl fs ; do
+    list_datasets | while read kl fs ; read cm fs ; do
         lock_fs "$fs"
     done
 }
A more sophisticated solution would make this a service(8) command so you can distinguish between service zfskeys start (regular boot‑up) and service zfskeys start_all_including_prompted_keys. The super‐duper implementation would allow me to enter prompted keys via ssh(1) at startup time.​
 
You must log in or register to reply here.
Back
Top