ZFS Reliability of ZFS native encryption

V

vidmak

First of all, this is not a question about ZFS-on-GELI vs. ZFS native encryption, which has been addressed many times exhaustively on this Forum. I understand the differences between the two conceptually and in terms of metadata visibility, performance, etc. I have also read Thread why-use-geli-underneath-zfs-these-days.97928, which contains some posts relevant to the topic at hand.

The background: recently, in order to guard myself better against data loss/corruption, I switched all of my home storage and backup infrastructure over from UFS-on-GELI to ZFS, with native encryption enabled for most datasets.

Unfortunately, the choice of native encryption was based on naïve enthusiasm and not altogether factually informed. In particular, I found out only afterward about the data corruption bug affecting non-raw zfs-send(8)/zfs-recv(8) that was (apparently) patched last year. Unfortunately, my backup scripts rely on this functionality for sending/receiving incremental snapshots, both locally and over the network. Nothing bad has happened, and I did patch my systems as per Security Advisory FreeBSD-EN-25:10.zfs. Yet reading further around the bug has been unnerving and brought to light, among other things, the relative lack of maturity of the ZFS native encryption code base.

Has anyone here actually had this bug rear its head under ordinary conditions, and if so, was it possible to recover from it, e.g. with a zpool-scrub(8)? Is it irrational in general to worry about this?

There must be tons of users out there who have been using ZFS native encryption at scale for years without any issues; if you are one of those people, it would be great if you could share your experience. Failing such reassurance, I am contemplating switching everything to ZFS-on-GELI. This would entail some initial pain, as I am rather locked in to native encryption at the moment, but would still be vastly preferable to risking data loss. My 2¢.
 
Don't take the silence of the forum as anything other than your thread going unnoticed by a billion possible reasons.

I have no direct experience, but according to my findings you are more than fine using the native encryption as long as you use the raw option of the zfs send command.

I know that I'm not precisely the kind of reassurance you are looking for, because I lack direct experience, but perhaps this comment will help to jump-start the thread (perhaps).
 
vidmak said:
There must be tons of users out there who have been using ZFS native encryption at scale for years without any issues; if you are one of those people, it would be great if you could share your experience. Failing such reassurance, I am contemplating switching everything to ZFS-on-GELI. This would entail some initial pain, as I am rather locked in to native encryption at the moment, but would still be vastly preferable to risking data loss. My 2¢.
Click to expand...

I don't think there a ton of users of ZFS encryption. It is a rather new feature and most install scripts (both FreeBSD and Linux) use an encrypted block layer under ZFS when asked to encrypt. ZFS native encryption has to my knowledge not received a serious review by security researchers, that can make it less attractive than e.g. cryptsetup on Linux.

I would say it is too late to switch to ZFS on GELI because of that bug that is now fixed.
 
Thanks to both of your for your replies, even if they do not entirely alleviate the concerns expressed above. As stated by cracauer@, the default installer option being ZFS over GELI is a weighty argument. Certainly enough to think about for the time being.
 
I have been using ZFS Encryption for some removable drives and a two VMs on laptops so far. No issues. Trying to send a zfs snapshot without the --raw parameter reliably gives me an error message, so I am not sure where the risk for data corruption might be.
 
ZFS encryption’s main benefit over GELI is that you can send encrypted (and incremental) data for backup onto a remote zpool managed by an untrusted third-party. The encrypted backup can be verified and managed (snapshots removed for example) without loading the encryption key, or even having it on the remote system.

If you don’t have this use case as a need, just stick with GELI for encrypted-at-rest data.
 
The other benefit is the ability to decrypt just those filesystems you want at a time, and then unload the key / make them inaccessible again. This is interesting for user home directories (decrypt only while logged in), for example, but last I looked some of the plumbing to make it useful isn’t quite ready yet.
 
rootbert said:
I can second that. My performance tests a while back showed that GELI was about 3x faster on average than the encrypted zfs dataset.
Click to expand...

Did you try with OpenZFS 2.4.0?

github.com

Release zfs-2.4.0 · openzfs/zfs

We are excited to announce OpenZFS 2.4.0! Supported Platforms Linux: compatible with 4.18 - 6.18 kernels FreeBSD: compatible with versions 14, 15 & 16 Key Features in OpenZFS 2.4.0: Quotas: Allo...
github.com github.com

> Better encryption performance using AVX2 for AES-GCM (#17058)
 
Eric A. Borisch said:
ZFS encryption’s main benefit over GELI is that you can send encrypted (and incremental) data for backup onto a remote zpool managed by an untrusted third-party. The encrypted backup can be verified and managed (snapshots removed for example) without loading the encryption key, or even having it on the remote system.

If you don’t have this use case as a need, just stick with GELI for encrypted-at-rest data.
Click to expand...
Encrypted at rest vs everything else. It sounds like ZFS encryption means the data is encrypted during transit of a ZFS send, which depending on the environment (Government, military, finance) could be a requirement. Kind of equivalent to "let me tar up a directory structure, encrypt it, then rsync and decrypt".

Eric A. Borisch said:
The other benefit is the ability to decrypt just those filesystems you want at a time, and then unload the key / make them inaccessible again. This is interesting for user home directories (decrypt only while logged in), for example, but last I looked some of the plumbing to make it useful isn’t quite ready yet.
Click to expand...
I think this is a "good idea" for multi user systems so I can't look at your stuff. But interesting timing on where it gets done. Perhaps different points when sitting at the physical system (session manager, cli) and remote access (ssh, RDP); almost seems to need tie in with "login". login: validate username password against system resource, then decrypt user home to run any .login/.profile SHELL stuff. Non-trivial at the corner cases.
 
From what I understand of the issue that was fixed (FreeBSD-EN-25:10.zfs), the problem was not in the encryption engine itself, but in the handling of non-raw send streams for encrypted datasets.

In other words, the AES-GCM encryption layer was not fundamentally broken. The bug was related to how zfs send/recv processed encrypted data when not using the --raw option. That distinction matters, because it means the core on-disk encryption logic and block checksumming were not compromised.

Importantly, this was not a case of silent pool corruption in the sense of random undetected bit rot spreading through the filesystem. ZFS still maintains block-level checksums independently of encryption. If data corruption occurs at the block level, a zpool scrub will detect checksum mismatches and, assuming redundancy (mirror/RAID-Z), will attempt to repair them.

So the realistic risk profile was tied to the send/receive stream correctness rather than to encryption silently corrupting stored blocks.

Using zfs send --raw for encrypted datasets is generally considered the safer and more semantically correct approach anyway, because it preserves the encrypted form of the data and avoids re-encryption or transformation during the send process.

If your systems are patched and you are using raw sends going forward, there is no strong technical reason to assume that native encryption is inherently unreliable. Its codebase is younger than the rest of ZFS, yes, but it shares the same core transactional and checksum infrastructure.

In short: the bug was real, it has been fixed, it did not imply a fundamentally broken encryption engine, and ZFS’s integrity model (checksums + scrub) still applies as expected.


 
rbranco said:
> Better encryption performance using AVX2 for AES-GCM (#17058)
Click to expand...
no, my tests were I would say somewhere around 2023 or maybe 2024.

Furthermore, from a security perspective, it was possible to retrieve metadata, which is a no-go for me: you can make an image of a zpool, alter the data on disk by changing random bytes of that disk and then make a scrub - zpool status would list all the files it had problems with (without entering the key)
 
rootbert said:
no, my tests were I would say somewhere around 2023 or maybe 2024.

Furthermore, from a security perspective, it was possible to retrieve metadata, which is a no-go for me: you can make an image of a zpool, alter the data on disk by changing random bytes of that disk and then make a scrub - zpool status would list all the files it had problems with (without entering the key)
Click to expand...

I couldn't reproduce with this:

Code: 
$ truncate -s 128M zfs.img
$ sudo zpool create testpool ./zfs.img
$ sudo zfs create -o compression=off -o encryption=on -o keyformat=passphrase -o keylocation=prompt testpool/crypt
$ sudo dd if=/dev/urandom of=/testpool/crypt/testfile bs=1M status=progress
$ sudo zpool export testpool
# Important!  Otherwise the testfile will be listed in `zfs status -v testpool`
$ sudo zfs unload-key testpool/crypt
$ dd if=/dev/zero of=zfs.img bs=1M count=20 skip=20 conv=notrunc
$ sudo zpool import -d . testpool
$ sudo zpool scrub testpool
$ sudo zpool status -v testpool
  pool: testpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 64K in 00:00:00 with 98 errors on Sun Feb 15 23:18:19 2026
config:

    NAME                     STATE     READ WRITE CKSUM
    testpool                 ONLINE       0     0     0
      /home/ricardo/zfs.img  ONLINE       0     0   215

errors: List of errors unavailable: permission denied

Did you actually run `zfs unload-key` ? Because `zfs export` is not enough.

And I couldn't see filenames with `strings zfs.img` on an uncompressed pool.
 
You must log in or register to reply here.
Back
Top