ZFS Mounting ZFS failed with error 6 during boot

Micski · May 19, 2025

On a machine with FreeBSD 13.5, that was originally prepared with GELI encryption and ZFS by the FreeBSD installer, the following error message appears during boot.

 Mounting from zfs:zroot/ROOT/default failed with error 6.

Loader variables:

  vfs.root.mountfrom=zfs:zroot/ROOT/default

Manual root filesystem specification:

mountroot>

The problem occurred after a second SSD was installed and the server was rebooted. The problem is probably caused by GELI, that is somehow no longer able to attach during boot. If the machine is booted from a FreeBSD USB memory stick, then GELI can attach each SSD, if ZFS imports the boot partition with the GELI encryption keyfile first.

Could this be caused, if GELI on the second SSD was configured with same passphrase as primary SSD, but without the requirement for the keyfile component? Could this be caused by the ZFS cache? Other possible reasons for this error message?

astyle · May 20, 2025

I'm assuming that OP upgraded to 13.5-RELEASE, rather than doing a fresh install... A quick googling pointed me to a bug report that might be helpful: PR 271989.

meaw229a · May 20, 2025

micski said:
On a machine with FreeBSD 13.5, that was originally prepared with GELI encryption and ZFS by the FreeBSD installer, the following error message appears during boot.

Mounting from zfs:zroot/ROOT/default failed with error 6. Loader variables: vfs.root.mountfrom=zfs:zroot/ROOT/default Manual root filesystem specification: mountroot>

Have a look at the following link. It may be helps. I had a zfs error 6 before and it was a label issue.

https://forums.freebsd.org/threads/...ailed-with-error-6-after-fresh-install.91349/

Micski · May 20, 2025

astyle said:
I'm assuming that OP upgraded to 13.5-RELEASE, rather than doing a fresh install... A quick googling pointed me to a bug report that might be helpful: PR 271989.

This is correct; Machine installed around 2018 and have been maintained and upgraded regularly for years.

Micski · May 20, 2025

meaw229a said:
Have a look at the following link. It may be helps. I had a zfs error 6 before and it was a label issue.

https://forums.freebsd.org/threads/...ailed-with-error-6-after-fresh-install.91349/

That's interesting. Thanks.I am not sure, if I am seeing the same situation, but at one point, I did see lines with "Sun Solaris" and incorrect labels mentioned. However, as stated, I can manually GELI attach each SSD and manually ZFS import each pool. Would that be possible with such label issue? I guess, it is hard to come up with clear answers on this issue. What baffles me is, how this issue could arise from such a trivial operation like adding extra storage into default GELI and ZFS configuration.

Micski · May 20, 2025

meaw229a said:
Have a look at the following link. It may be helps. I had a zfs error 6 before and it was a label issue.

https://forums.freebsd.org/threads/...ailed-with-error-6-after-fresh-install.91349/

I noticed, that according to the manual for zpool-labelclear, it removes ZFS label information from the specified device, but the device must not be part of an active pool configuration. In my case, the added, and never used before, secondary SSD was merged with the primary SSD so both provide storage to the default zpool. This could mean, that I would need to un-merge and attempt to reverse the operation, before clearing or setting labels.

T-Daemon · May 20, 2025

Please tell us about the system setup. Is it a geli encrypted Root-on-ZFS partition on disk0-old, a encrypted / unencrypted Root-on-ZFS + encrypted pool2 on separate partition on disk0-old+ encrypted partition on disk1-new, added to pool2?

Where is the keyfile stored?

As I understand
geli provider disk0-old initialized passphrase + keyfile
geli provider disk1-new initialized passphrase

From the FreeBSD image stick

Code:

gpart show -lp

geli attach <all provider>

geli list

zpool import

cat /path/to/imported/zroot-pool/boot/loader.conf

Additional try: Boot system, at the FreeBSD boot menu "Escape to loader prompt"

Code:

load  /boot/kernel/kernel
load  /boot/kernel/geom_eli.ko
load  /boot/kernel/zfs.ko
boot

Micski · May 20, 2025

Before the change, there was a single SSD with GELI and ZFS storage pool zpool. After the change, there are two SSDs with GELI and stil one ZFS storage pool zpool. I am unsure, if the -b option should have been used, when preparing the secondary SSD.

I retrieved a copy of the loader.conf from a backup. This confirms the use of a keyfile on the unencrypted boot partition on the primary SSD. The secondary SSD does not use a keyfile, because I did not know, that any keyfile was in use at time of preparing the secondary. It seems, that older FreeBSD 11 or 12 installation generated a keyfile in addition to the user set passphrase.

sh:

# cat loader.conf
geli_ada0p4_keyfile0_load="YES"
geli_ada0p4_keyfile0_type="ada0p4:geli_keyfile0"
geli_ada0p4_keyfile0_name="/boot/encryption.key"
aesni_load="YES"
geom_eli_load="YES"
geom_eli_passphrase_prompt="YES"
vfs.root.mountfrom="zfs:zroot/ROOT/default"
kern.geom.label.gptid.enable="0"
zpool_cache_load="YES"
zpool_cache_type="/boot/zfs/zpool.cache"
zpool_cache_name="/boot/zfs/zpool.cache"
zfs_load="YES"
pf_load="YES"
pflog_load="YES"

Micski · May 22, 2025

To future readers, that might land here with similar issue: A kind member of the FreeBSD forum on Discord mentioned, that the GELI flag "-b" has to be set on each encrypted partition, that will be participating in the final ZFS storage pool. The flag was set, using a rescue USB live boot environment, and the error message disappeared. Note, that after this change, the ZFS cache also needed to be regenerated and an issue with a nested boot partition remained unsolved, but probably unrelated to this issue. I recommend, that a fresh install is considered, when storage is added to an older FreeBSD machine. It might be faster, more straight-forward and the result is a machine with current layout of partitions and file system.

astyle · May 22, 2025

micski said:
I recommend, that a fresh install is considered, when storage is added to an older FreeBSD machine.

One way to read that, it's either a bug or a missing feature - to consider the existing set of flags on a given ZFS pool/dataset and make sure it gets applied correctly to newly added storage.

But IIRC, GELI is a separate animal/utility. Based on that, I see another way to read this thread: Maybe have some good notes on how the original storage was created, what settings were applied, and so forth. Original settings do affect how you do things down the line. If you have good, detailed notes on how you originally set up your storage, adding more storage later will be easier. (Yeah, that is more work up front, I learned that the hard way on a different project.)