ZFS Modern setup of Disks + GELI + ZFS

0

0nighthawk0

Hi all,

Running FreeBSD 12.3.
Got a question on the recommended way to setup a GELI-Encrypted ZFS pool.

Basic info about requirements:
  1. These disks are just for a ZFS Storage Pool - they are not and will never need to be bootable.
  2. They will also never need to be compatible with another system - they will remain on FreeBSD (or a derivative).
  3. If the motherboard dies I need to be able to lift the disks to a new FreeBSD system and have them work.
  4. I don't particularly want to use the new dataset encryption on ZFS in 13.0 - although I understand some people will.
When i first did this years ago (like 12 or more years ago) I don't recall the initial setup exactly, i've obviously placed this somewhere so safe i can't find it, however the process to access the pool was similar to this:

Boot system.
System boots fine.
SSH in to attach/decrypt drives.
Restart zfs service
Pool available.
Also the vdev's in pool were tied to "glabels" which were just part of the serial number of the disk.

I quite liked this process as it enabled me to boot the system but still keep the storage secure.

Now while this process might seem vague, at the time I remember having to align the sectors on the disk etc and did quite a bit of performance tuning, i remember the glabel label went in a certain area of the drive and geli occupied either all of it, or most of it minus a sector .... or maybe it was full disk and labelled inside the .eli? No recollection.

My original pool was originally i want to say v7 or 8? then v15, then v23?? eventually I upgraded it to feature flags version 5000.
I copied all my data off elsewhere and dont have the original machine to reference.

I have got a fresh(ish) system and just need to build a platform that can be updated for the next 5-10 years.
So there is no existing pool, nothing that can't be done from scratch.

So just looking for the modern way to set this up.
I've searched all over and there are so many conflicting views, but all from a myriad of different years - so I guess i'm polling views here for the recommended method now.
 
To create a geli(8) encrypted provider for a ZFS storage pool is straitforward.

Utilities:
  • gpart(8) create partition scheme and partition on disks
  • geli(8) init (initialize) the providers (partitions)
  • geli(8) attach providers
  • zpool(8) create storage pool
Example for a three disk mirror pool:
Code: 
Destroy old partition table:
# gpart destroy -F ada0

# gpart create -s gpt ada0
# gpart add -t freebsd-zfs -a 1m  ada0

(Repeat for all disks. If set labels here (-l <label>) use labels for following commands.)

# geli init -l 256 -s 4096 ada0 ada1 ada2
(enter passphrase)

# geli attach ada0 ada1 ada2
(enter passphrase)

# geli status   # shows providers

# zpool create tank mirror ada0.eli ada1.eli ada2.eli

Don't forget to backup the metadata for the providers on multiple, external storage devices, mark the devices accordingly, and store them in secure, rememberable places.

For additional options and details for gpart(8), geli(8), zpool(8) see manuals.

0nighthawk0 said:
I remember having to align the sectors on the disk etc and did quite a bit of performance tuning, i remember the glabel label went in a certain area of the drive and geli occupied either all of it, or most of it minus a sector .... or maybe it was full disk and labelled inside the .eli? No recollection.
Click to expand...
I'm not aware of the necessity to align sectors on disks. Certainly there is no mention in the manuals.
 
please, notice a common language misconception about GELI and encrypted ZFS pools - pools are on top of lower level encryption, FDE ( full-disk-encryption ) which is device blocks. ZFS is then created on already encrypted device.
 
angry_vincent said:
please, notice a common language misconception about GELI and encrypted ZFS pools - pools are on top of lower level encryption, FDE ( full-disk-encryption ) which is device blocks. ZFS is then created on already encrypted device.
Click to expand...

Thanks,

Prompts a question though.

Is there a reason that people don't just GELI encrypt the entire drive?
I understand that GELI metadata will make this slightly smaller than the full drive but basically negligible.

Then you mount the .eli as whatever device it was found as e.g. da2 etc - may play havoc with scripts if you chop and change a lot

Then you could gpt or glabel inside the mounted da2.eli.

Then tie the zfs to a gpt or glabel created inside the .eli, when naming the zfs virtual devices?

Everything seems to occur in the opposite order from every article I have read and although what T-Daemon describes is a nice concise method that works (as i've just tested it), i'm just wondering why it isn't done that way around?

I'm not sure what complexity or performance difference this would introduce or if you are just asking for trouble at this point.

Its not necessarily more secure or anything other than not exposing drive labels outside the .eli, I just wondered why it wasn't done.
Is it to avoid something overwriting the geli metadata? sector alignment with the phyisical hardware?

Think i might have answered my own question as its would be quite difficult to know which drive might have been da2.eli for purpose of metadata restore.
 
what i meant ( and only this :) ) is usage of language such as 'ZFS encrypted pools', which you can find in many guides, GELI does not encrypt them in any way but rather encrypts your drive. since, OpenZFS-2, there also such thing as native encryption of pools - not to be confused with disk encryption, which what GELI does. There is no possibility ( not implemented yet ) to have boot from pool with "native encryption"
 
0nighthawk0 said:
Is there a reason that people don't just GELI encrypt the entire drive?
I understand that GELI metadata will make this slightly smaller than the full drive but basically negligible.

Then you mount the .eli as whatever device it was found as e.g. da2 etc - may play havoc with scripts if you chop and change a lot

Then you could gpt or glabel inside the mounted da2.eli.

Then tie the zfs to a gpt or glabel created inside the .eli, when naming the zfs virtual devices?

Everything seems to occur in the opposite order from every article I have read and although what T-Daemon describes is a nice concise method that works (as i've just tested it), i'm just wondering why it isn't done that way around?
Click to expand...
I can't speak for others, on my side it's force of habit.

Dealing more with full disk encrypted Root-on-ZFS installations than with pure data disks, the FreeBSD system to be bootable expects on the disk a partition table, boot partition(s) (BIOS, UEFI, UEFI+BIOS), and the geli(8) provider on a separate partition.

For pure data disks it's perfectly fine to create from the whole disk a geli(8) provider first:
Code: 
# geli init ... provider(s), i.e. da0, ...
# geli attach provider(s)
# gpart create partition table, add partition+label on provider(s).eli
# zpool create ... gpt/name0 gpt/name1 gpt/name2 (nameN being partition label, labeled with gpart on provider(s).eli)

0nighthawk0 said:
I'm not sure what complexity or performance difference this would introduce or if you are just asking for trouble at this point.
Click to expand...
I can't imagine any complexity or trouble here, but I can't rule out problems with a unusual setup.

Regarding performance, I haven't done any benchmarking. I don't have that much data to transfer from one device to another (e.g. incremental backups). The question of performance didn't arise in my case. My use case is a home user case with a few TB's.

0nighthawk0 said:
Its not necessarily more secure or anything other than not exposing drive labels outside the .eli, I just wondered why it wasn't done.
Is it to avoid something overwriting the geli metadata? sector alignment with the phyisical hardware?

... as its would be quite difficult to know which drive might have been da2.eli for purpose of metadata restore.
Click to expand...
Not at all. If glabel(8) is used to label the disk(s),e.g. da0, the metadata backup file will be saved with the disk label prefixed in the default directory /var/backups/.

Or initialize the provider ( geli init) with the "-B" option set, to specify the place to save the file.

In case of multiple disks being initialized in one command, use "PROV" in file name (see geli init -B for details). You need to create a "label" directory first when using glabel(8)ed disks though, e.g.:
Code: 
# mkdir -p /root/geli/label
# geli init -l 256 -s 4096 -B /root/geli/PROV.eli label/<name1> label/<name2> label/<name3>
 
Last edited:
T-Daemon said:
Example for a three disk mirror pool:
Code: 
Destroy old partition table:
# gpart destroy -F ada0

# gpart create -s gpt ada0
# gpart add -t freebsd-zfs -a 1m  ada0

(Repeat for all disks. If set labels here (-l <label>) use labels for following commands.)

# geli init -l 256 -s 4096 ada0 ada1 ada2
(enter passphrase)

# geli attach ada0 ada1 ada2
(enter passphrase)
...
Click to expand...
You're creating partitions, yet using geli on the raw device instead of the partition. Shouldn't it be geli init -l 256 -s 4096 ada0p1 ada1p1 ada2p1 and geli attach ada0p1 ada1p1 ada2p1?
 
You must log in or register to reply here.
Back
Top