ZFS Is there a way to remove geli encryption from a raidz2 array without losing data or using additional devices?

[puppydog]

I have a FreeBSD 13.2-RELEASE server with a single six-disk ZFS raidz2 array which is encrypted with geli. For certain reasons I would like to completely remove the encryption from this array, but I need to do so without using any disks other than the six already in use, and without losing data. All disks in the array are healthy.

It seems like there must be a way to do it by removing a disk from the array, reformatting it without encryption, then rejoining it to the array, resilvering, and repeating until all disks have been cycled through. But I'm not sure if that's possible, or if there's a simpler way to do it. Or for that matter, how to even go about doing that safely.

If it's possible, how do I go about doing this?

Array:

root@server:~ # zpool status
  pool: data
 state: ONLINE
  scan: resilvered 2.85M in 00:00:01 with 0 errors on Tue Apr 11 19:35:15 2023
config:

    NAME             STATE     READ WRITE CKSUM
    data             ONLINE       0     0     0
      raidz2-0       ONLINE       0     0     0
        mfid0p4.eli  ONLINE       0     0     0
        mfid1p4.eli  ONLINE       0     0     0
        mfid5p4.eli  ONLINE       0     0     0
        mfid2p4.eli  ONLINE       0     0     0
        mfid3p4.eli  ONLINE       0     0     0
        mfid4p4.eli  ONLINE       0     0     0

errors: No known data errors

 

[cracauer@]

That is possible only if you have 6 separately encrypted disk devices.

 

[mer]

That is possible only if you have 6 separately encrypted disk devices.

Does his output of zpool status imply that?

 

[monwarez]

You should be able to do this with

Perform a backup of the data (using zfs replication, so you will need at least another machine/disk for it).

zpool detach data mfid0p4.eli
geli detach mfid0p4.eli
Delete the partition, and re-create it just in case
zpool replace data mfid0p4.eli mfid0p4
Do a resilver just in case.
And repeat.

 

[T-Daemon]

It seems like there must be a way to do it by removing a disk from the array, reformatting it without encryption, then rejoining it to the array

It's possible without removing or detaching a disk (which are not supported on a raidz2 anyway).

I've just tested it in a VM, boot the system into multi-user mode, apply following steps for each disk (replace "X" with the same disk number):

# geli kill mfidXp4
# zpool replace data mfidXp4.eli mfidXp4

I suggest you create a virtual machine first, same setup, and practice there a few times.

Make sure there is a backup, just in case.

 

[cracauer@]

Does his output of zpool status imply that?

Yes, if I read that correctly you have individual GELI volumes for each disk. What @monwarez posted should work.

 

[T-Daemon]

What @monwarezposted should work.

zpool detach data mfid0p4.eli

zpool-detach(8) is not supported on a raidz2.

The manual clearly states:

NAME
     zpool-detach – detach device from ZFS mirror

 

[cracauer@]

Yes, sorry, you need zpool-offline. This has been a pain point for me in the past.

If you have hotswap it might be best to physically remove the drive.

 

[puppydog]

T-Daemon's suggestion of geli kill followed by zpool replace seems to be working. My first disk is resilvering now, and it appears to be going well. Looks like it's taking about a day per disk and I don't want to risk doing two at once, so hopefully by next weekend I'll be able to test that everything turned out well. Thanks

 

[mer]

I've done similar things with increasing the size of a mirror and all the recommendations I found said "one disk at a time. let it resilver completely and then do the next one".