ZFS GELI vs ZFS encrypted dataset

J

jbo@

Developer
I've been using ZFS encrypted datasets pretty much since the first day of availability. On this forum and also the mailing lists I see a lot of people running ZFS "on top of" GELI.
I'd like to ask: Is that usually more of a legacy thing or are there situations where one would prefer ZFS on GELI rather than just using native ZFS encrypted datasets?
 
Encrypted ZFS has only been available since the import of OpenZFS with 13.0. Before that the only way to get an encrypted volume was by leveraging geli(8) and putting ZFS on top of that.

jbodenmann said:
Is that usually more of a legacy thing or are there situations where one would prefer ZFS on GELI rather than just using native ZFS encrypted datasets?
Click to expand...
One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
 
Jup - I get that. Hence my question: Is using ZFS on top of GELI mostly a legacy thing or are there situations where one would prefer to use that over encrypted ZFS (assuming that it is available)?
 
So far I'm preferring the ZFS encryption. But it has a few drawbacks. Even if the dataset is locked you can still see that it exists. In that respect GELI+ZFS was 'better' as it also blocked access to the entire ZFS structure.
 
Can you boot from a ZFS-encrypted root partition? I think that's only possible with GELI.

Also, I remember reading that GELI would be faster, but I can't speak from personal experience (never used ZFS encryption).
 
bsduck said:
Can you boot from a ZFS-encrypted root partition? …
Click to expand...

I suspect not. I mean, I should not expect the loader to present a prompt to enter a passphrase; <https://cgit.freebsd.org/src/log/?qt=grep&q=loader&h=main> no mention of zfs on the first page.


What's below is essentially fixed, however this is for the zfskeys service script – long after boot:
– from /var/log/console.log yesterday, with 50e244964e9:

Code: 
Apr  7 01:17:35 mowa219-gjp4-8570p-freebsd kernel: Key file /usr/home/grahamperrin/Documents/personal/VirtualBox.key not found, empty or unreadable. Skipping Transcend/VirtualBox..
Apr  7 01:17:35 mowa219-gjp4-8570p-freebsd kernel: Loading key for Transcend/VirtualBox from file:///usr/home/grahamperrin/Documents/personal/VirtualBox.key..

– essentially fixed, however the first of those two lines is debatably spurious (the file that's reportedly not found is found a split-second later).
 
SirDice said:
Encrypted ZFS has only been available since the import of OpenZFS with 13.0. Before that the only way to get an encrypted volume was by leveraging geli(8) and putting ZFS on top of that.


One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
Click to expand...
Hi SirDice,
This is surprising to read, in one of the last journals of freebsdfoundation they were still saying geli and zfs is OK. The issue you mention about the bad sectors, I though this was exactly what ZFS was doing, keeping track of bads ones and never use them again. But if I understand correctly GELI is the issue here. And just to be clear I believe you, but gives me the feeling I think my data is safe but it's maybe not at all.

Personally I learned a lot from ZFS by having situations where failures or crashes occurred. The more crashes the better, then I learn more. With FreeBSD on virtualbox on another OS, the chance of crashes are higher. But I wonder, how exactly do you know that the sector is bad? Is there a command in zfs or with zdb that can give you all bad sectors?

And coming back to the geli-zfs story, does it mean in the future ideally we better move to native encryption?

Thanks in advance!

Best Regards,
R
 
Let me comment a bit on the two encryption approaches. Both are still very viable today, but serve a different kind of audience.

GELI + ZFS is a full disk encryption scheme, 0 information is leaked to the attacker that would insert said hard drive in its own computer. GELI + ZFS is perfectly fine to keep ZFS working as it should, fixing bad sectors, check summing, etc. I think GELI has the option to also enable that feature but that would collide with ZFS, so only use GELI for the encryption part. You can boot from it (support in the bootloader is present).

ZFS native encryption is a dataset encryption scheme (not full DISK/GEOM), you can google a bit about this but it does leak certain metadata, like dataset names, sizes of files?? (don't quote me on the latter) etc. Depending on your use case, one will be better suited than the other. For our home server applications, typically, ZFS native encryption is fine. The bootloader has not been modified yet to boot from ZFS native encryption. It has been modified to support booting from a ZSTD compression enabled root (both are OpenZFS features).
 
Last edited:
driesm Thank you for summarizing/comparing those two methods.

driesm said:
The bootloader has not been modified yet to boot from ZFS native encryption.
Click to expand...
The "yet" part seems particularly interesting to me. Is there any evidence that would result in reasons to believe that in the future we'll be able to boot FreeBSD from an encrypted ZFS dataset (where "encrypted ZFS dataset) is referring to a ZFS dataset that was encrypted through ZFS's native encryption capabilities)?
 
  • Thanks
Reactions: veg
SirDice said:
One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
Click to expand...

This should not be a problem.
Weren't we running GEOM NOP devices for sector size translation back 10 years ago when a bunch of WD Green drives were "misreporting" sector size?

P.S. there are a lot of GELI-pros but one con would be that ZFS is portable and GELI isn't (in practice). If your storage setup depends on a GELI feature then it's bound to FreeBSD versions that implement that feature. For 99.9% of our cases this is irrelevant but it's still worth to mention.
 
jbodenmann said:
… reasons to believe that in the future we'll be able to boot FreeBSD from an encrypted ZFS dataset …
Click to expand...

Whilst the bootloader has not yet gained support, it should (I guess) be possible to begin with an encrypted pool/ROOT, then install FreeBSD the long way, then whenever you need to boot a boot environment e.g. pool/ROOT/default: first, boot from something that allows you to unlock the ROOT. That thing might be a minimalist installation of FreeBSD on a USB flash drive.

That's clumsily worded, but hopefully enough for people to get the idea. Not to say that it should be done (obviously, it's smoother to use GELI at the moment), but it probably can be done, in theory.

Postscript

grahamperrin said:
FreeBSD bug 263171 – loader(8) and boot loader menu support for boot with OpenZFS-encrypted ROOT
Click to expand...

Holistically, support for OpenZFS-encrypted ROOT should be added to bsdinstall(8), however I don't imagine this happening. Gut feeling: it'll be a feature of a modernised installer (not bsdinstall).

More realistic (than bsdinstall): 263234 – Add support for OpenZFS encryption to adduser.
 
grahamperrin@ said:
Holistically, support for OpenZFS-encrypted ROOT should be added to bsdinstall(8), however I don't imagine this happening. Gut feeling: it'll be a feature of a modernised installer (not bsdinstall).
Click to expand...
Has this effort started, and if so, would anyone know where to best track its progress, by any chance?

In the meantime, I’d love to figure out what’s considered best practice for manually (until an installer implements it) setting up boot on a natively encrypted root dataset.

Cheers!
 

Attachments

  • yes-YES.png
    yes-YES.png
    827.3 KB · Views: 123
You must log in or register to reply here.
Back
Top