ZFS Is this a correct way to enable native ZFS encryption on a dataset?

[puppydog]

I'm wanting to use native ZFS encryption with one of my datasets and all of its child datasets, and I wanted to make sure I'm going about this correctly.

It's a large amount of data (something in the area of 30 TB) but fortunately I do have enough space to temporarily have the data all doubled. For this example, the dataset to be encrypted is at pool/data and there are two lower datasets at pool/data/one and pool/data/two. My tentative plan then, is to do this:

1. Rename the dataset (this should affect the two child datasets too?):
# zfs rename pool/data pool/data-unenc
2. Snapshot the dataset:
# zfs snapshot -r pool/data-unenc@encryptionprep
3. Send the unencrypted dataset and receive it as a new dataset with encryption and all desired properties set:
# zfs send -R pool/data-unenc@encryptionprep | zfs receive -o encryption=on -o keyformat=passphrase -o compression=lz4 -o (and so forth for every property I want to set) pool/data
4. Delete the unencrypted dataset:
zfs destroy -r pool/data-unenc
5. zeroize free space with dd:
dd if=/dev/zero of=/data/bigfile.dd;rm /data/bigfile.dd

Is my basic plan here correct? Especially with step 3, what I'm aiming for is by the time the receive is done, /pool/data and all of its child datasets exist exactly as they did before I renamed the original dataset, except now it's encrypted.

Am I approaching this in a valid way? Or should it be done differently?

 

[Mirror176]

1. Only snapshots can be recursively renamed (-r). Your goal requires performing the step multiple times.
5.a. Writing zeros to a file on a dataset with compression is likely not what you meant; disable compression before doing so and reenable when done.
5.b. You probably want to zero free space using a sequence like drive removal, blank the drive, replace in pool and resilver, move to next drive when complete. Writing zeroes opens up questions of if reserved space got filled/overwritten (hint: some space has to be reserved as a COW filesystem cannot write about a delete if it is full). ZFS history also is not cleared by these steps. I'm not sure of the effect of filling a pool beyond capacity causing negative impacts for performance and fragmentation; you will be deleting the file when done, but 'any' writes performed while full will be in that lower performing state so if not part of the file they will remain in suboptimal placement after delete.
5.c. One solution knowingly doesn't leave hiding places: backup data, delete pool, recreate pool.
5.d. ZFS encryption I thought was still considered experimental. FreeBSD has geom's geli or gbde (mabye gvinum/vinum had stuff too; I forget nwo) as encryption options that ZFS could be placed inside of. I think the installer uses geli if encryption is enabled.

 

[Mirror176]

Writing /dev/random instead of /dev/zero removes a lot of the need to disable compression; sometimes random characters can have a pattern and the compressor will still be trying before it, hopefully quickly, aborts when not enough is going to be saved. Filling a live filesystem in 'any' case is still just a bad idea; single user mode would limit other processes trying, and failing, to write files at the relevant time. The delete will not immediately return free space (write queues and whatnot) making that a larger window of possibility. Not all transaction group blocks are immediately overwritten so I'd wonder if old blocks are still there and protected for ZFS's resiliency and if some blocks may contain file data instead of pointing to it (small files) could hold some things longer.

 

[Alain De Vos]

Note , you can do "zfs unload-key -r ZPOOL/encrypted".

 

[puppydog]

1. Only snapshots can be recursively renamed (-r). Your goal requires performing the step multiple times.

In this case since I'm just renaming the parent dataset, all the child datasets changed appropriately.

5.d. ZFS encryption I thought was still considered experimental. FreeBSD has geom's geli or gbde (mabye gvinum/vinum had stuff too; I forget nwo) as encryption options that ZFS could be placed inside of. I think the installer uses geli if encryption is enabled.

It is, but in this case for certain reasons I have a requirement that the pool exist on raw disks and only zfs encryption be used. I actually just finished removing geli encryption from all the disks in the pool.

Filling a live filesystem in 'any' case is still just a bad idea; single user mode would limit other processes trying, and failing, to write files at the relevant time.

This is just a big data storage pool so this isn't a concern. It'll be fine if it's filled to 100% capacity for a brief period.

5.a. Writing zeros to a file on a dataset with compression is likely not what you meant; disable compression before doing so and reenable when done.

Good point, I didn't think of that.

5.b. You probably want to zero free space using a sequence like drive removal, blank the drive, replace in pool and resilver, move to next drive when complete.

This would be the preferable approach if it worked. At first my thought was that the resilvering process would bring back any remnants from the unencrypted pool, but thinking more on it that might not be the case. I might do exactly that once the pool is in the state I want.

 

[wolffnx]

I dont want to disrespect you, but I think that you are making a easy process too long...
If I understood well , you want to create one encrypted dataset?
for example , your pool is called "zroot"

zfs create -o encryption=on -o keyformat=passphrase zroot/encrypted
enter the passphrase 2 times

for testing..

umount /zroot/encrypted
zfs unload-key zroot/encrypted

zfs load-key zroot/encrypted
[enter passhrase]
mount /zroot/encrypted

 

[puppydog]

I dont want to disrespect you, but I think that you are making a easy process too long...
If I understood well , you want to create one encrypted dataset?
for example , your pool is called "zroot"

zfs create -o encryption=on -o keyformat=passphrase zroot/encrypted
enter the passphrase 2 times

for testing..

umount /zroot/encrypted
zfs unload-key zroot/encrypted

zfs load-key zroot/encrypted
[enter passhrase]
mount /zroot/encrypted

No, the goal was to move over 30 TB of existing data in multiple nested unencrypted datasets into new datasets that are identical in every way except for having native encryption enabled. I'm still waiting on step 3 in my outline to finish which I expect to be done sometime overnight tonight, so I'll be able to check in the morning if my plan worked.

 

[Mirror176]

The difficulty was without leaving unencrypted data behind. COW filesystem tries to create before delete when migrating so it brings up concern that old unencrypted data still exists on disk obviously as free space and less obviously possibly as historical blocks that have not yet been fully cycled out of the ZFS structure.

 

[wolffnx]

No, the goal was to move over 30 TB of existing data in multiple nested unencrypted datasets into new datasets that are identical in every way except for having native encryption enabled. I'm still waiting on step 3 in my outline to finish which I expect to be done sometime overnight tonight, so I'll be able to check in the morning if my plan worked.

now I understand, good luck

 

[ThePowerOfFuet]

The difficulty was without leaving unencrypted data behind. COW filesystem tries to create before delete when migrating so it brings up concern that old unencrypted data still exists on disk obviously as free space and less obviously possibly as historical blocks that have not yet been fully cycled out of the ZFS structure.

This is what `zpool trim` (or, failing that, `zpool initialize`) is for.

 

[Mirror176]

`zpool trim` only works on disks supporting trim but yes `zpool initialize` would cover the rest. If taking a dd of a drive that will be compressed then initializing it beforehand is a good approach; haven't attempted that in a long time but I hope I remember 'initialize' is available if I find myself doing it again. Deterministic Read After TRIM (DRAT) and similar describe what to expect of a trimmed block being read; not all will return zeroes within that space even 'if' they do not return previous data. That starts getting down the route of https://utcc.utoronto.ca/~cks/space/blog/tech/SSDBlockDiscardHowSecure but if security was the reason for initiating it then -d on trim should be used when supported. Unfortunately I haven't seen a trim option that controls the value that will be read back if trying to control it for compression so secure is likely best chance there too.

Those commands only operate on free space but still leave the potential for data in zfs blocks that have not been fully released. They have to exist by design as that is how there is somewhere to go back to when a future write started but didn't complete successfully yet. zpool import has a rollback feature that uses this so it does exist as something to be aware of, even if it will end up being short lived. If security is the goal, it is good to think anout any 'if' instead of the obvious user controlled things like 'checkpoints maintaining data despite a destroy should be removed to release those blocks.'