ZFS trying encryption with geli

[PMc]

Around April 2009, that is now ten years ago, I had a look at the then rather new GEOM tools, and already then noticed the geli(8) as a nicely crafted piece of software.
At that time most of my drives were partitioned with gvinum(8), and I figured that both would not work nicely together - because the GEOM tools could not properly know if they should grab the plex, the subdisk or the volume from gvinum. So, if a configuration might succeed, it would be hosed after next boot, at the latest.

Nowadays, almost all of my drives are in ZFS. Furthermore, recent political developments may make it desireable to have encrypted disks - so I concluded I might give the matter a new try. Also I imagined that over the course of ten years the matter may have ripened and might work out of the box by now.

Long story short: it doesn't.

After I had made a proper plan, and then started to convert the first one or two disks, I got the idea that I might try a reboot, to see how much of the configurations would survive that. And then, the issues sprang into my face like alien facehuggers.

1. Only the swap partition is autodetected. ZFS partitions are not. I suppose these have to be put into /etc/rc.conf, where there is a variable geli_devices.
2. When done so, the devices will be activated at boot, each for it's own. This means I have to enter the passphrase again and again, for each and every log device, spare device, cache device, raid device etc.etc.etc - some thirty or fifty times. This is not funny, and furthermore, due to that PKCS-wtf stuff it takes a significant amount of time.
   It seems, people handle this issue by using keyfiles instead of passphrases, so these keyfiles can be configured in rc.conf and automatically applied on boot. I fail to get the point in that. I don't see any point in leaving my house, closing the door, locking the door and then attaching the key with duct-tape on the door, either. Keyfiles seem to work in exactly that fashion, and under such circumstances I would abandon encryption and put that electricity to better use.
3. At the time when geli activates these devices, ZFs has already collected it's drives - and marked the encrypted (and therefore not yet existng) ones as UNAVAIL.
4. If one tries to fix the havoc and online the devices manually, that doesn't work
   

   # zpool online gr ada1s3.eli
   warning: device 'ada1s3.eli' onlined, but remains in faulted state
   use 'zpool clear' to restore a faulted device

   The reason seems to be that there is an option in the rc.conf defaults geli_autodetach="YES". So after ZFS looks onto that device (and, as it seems, does not keep it open continuously), it is gone.

Now, most of this seems not to be real problem - it should be solveable with some lines of scripting. But for quite a couple of years I now notice that many people are using ZFS, and many people are using encryption - so it surprizes me that no common agreement on how to do it has found it's way into the code of the OS, and everybody seems to hack their own solution.

 

[linux->bsd]

It seems, people handle this issue by using keyfiles instead of passphrases, so these keyfiles can be configured in rc.conf and automatically applied on boot. I fail to get the point in that. I don't see any point in leaving my house, closing the door, locking the door and then attaching the key with duct-tape on the door, either. Keyfiles seem to work in exactly that fashion, and under such circumstances I would abandon encryption and put that electricity to better use.

Is it not possible to store those keyfiles on the encrypted zroot dataset? To follow your analogy, you'd be locking up your house key in a password-protected box attached to your front door. You'd be prompted at the bootloader to enter your zroot geli device password, and the rest should be attached during the boot phase I would expect.

 

[SirDice]

Is it not possible to store those keyfiles on the encrypted zroot dataset?

This would present a chicken and egg problem. To access the encrypted data you need the key, that key is stored inside the encrypted data which can only be unlocked with the key, that's stored inside the encrypted data.

 

[D-FENS]

Around April 2009, that is now ten years ago, I had a look at the then rather new GEOM tools, and already then noticed the geli(8) as a nicely crafted piece of software.
At that time most of my drives were partitioned with gvinum(8), and I figured that both would not work nicely together - because the GEOM tools could not properly know if they should grab the plex, the subdisk or the volume from gvinum. So, if a configuration might succeed, it would be hosed after next boot, at the latest.

Nowadays, almost all of my drives are in ZFS. Furthermore, recent political developments may make it desireable to have encrypted disks - so I concluded I might give the matter a new try. Also I imagined that over the course of ten years the matter may have ripened and might work out of the box by now.

Long story short: it doesn't.

After I had made a proper plan, and then started to convert the first one or two disks, I got the idea that I might try a reboot, to see how much of the configurations would survive that. And then, the issues sprang into my face like alien facehuggers.

1. Only the swap partition is autodetected. ZFS partitions are not. I suppose these have to be put into /etc/rc.conf, where there is a variable geli_devices.
2. When done so, the devices will be activated at boot, each for it's own. This means I have to enter the passphrase again and again, for each and every log device, spare device, cache device, raid device etc.etc.etc - some thirty or fifty times. This is not funny, and furthermore, due to that PKCS-wtf stuff it takes a significant amount of time.
   It seems, people handle this issue by using keyfiles instead of passphrases, so these keyfiles can be configured in rc.conf and automatically applied on boot. I fail to get the point in that. I don't see any point in leaving my house, closing the door, locking the door and then attaching the key with duct-tape on the door, either. Keyfiles seem to work in exactly that fashion, and under such circumstances I would abandon encryption and put that electricity to better use.
3. At the time when geli activates these devices, ZFs has already collected it's drives - and marked the encrypted (and therefore not yet existng) ones as UNAVAIL.
4. If one tries to fix the havoc and online the devices manually, that doesn't work
   

   # zpool online gr ada1s3.eli
   warning: device 'ada1s3.eli' onlined, but remains in faulted state
   use 'zpool clear' to restore a faulted device

   The reason seems to be that there is an option in the rc.conf defaults geli_autodetach="YES". So after ZFS looks onto that device (and, as it seems, does not keep it open continuously), it is gone.

Now, most of this seems not to be real problem - it should be solveable with some lines of scripting. But for quite a couple of years I now notice that many people are using ZFS, and many people are using encryption - so it surprizes me that no common agreement on how to do it has found it's way into the code of the OS, and everybody seems to hack their own solution.

For autodetection at boot time you need to set the geli flags "g" and "b" for an encrypted boot+root partition. I use a fully encrypted disk where only the tiny unencrypted bootloader is stored without geli.
By setting "b" and "g" you will be asked for password at boot time and geli will try to mount the partition as an encrypted root.

This works if your drive is encrypted only with password and without a key file. Otherwise, you need to make the key file available to the boot loader - either by placing it on an unencrypted partition, or by encrypting its partition only with password and setting the "g" geli flag.
The key could be on a removable drive for example, if you want to carry it separate from the machine.

Also - setting the geli flags can be done with geli configure, but make sure the volume is detached first. Trying to set flags on an attached volume does not work BUT it does not show any errors. So I scratched my head quite a few times because of that.

P.S. I use GPT. I am not aware what's the procedure for MBR.

 

[PMc]

It seems, the keyfile solution is devised in the fashion like a car is handled: you have the car in the parking-lot, and you have the car-key on your keyring. Here you would have some usb-stick or such on the keyring. And in order to start the thing, you need to plug in the key, and when it has started, you can withdraw the key again. There are certainly some usecases where this can make sense. But I think it only disperses the risk: instead of stealing the machine only, the attacker needs to steal the machine and the key - that's a gradually harder difficulty, but not an essentially different difficultiy.

Then, if you decide to encrypt that keyfile with a personal password, so that if stolen it cannot be used, what kind of improvement is this over just encrypting the disk with a personal password? It seems many people do it this way, but I do not see the advantage (given that the design of the scheme is good enough to transform that password into properly random key-material for the actual encryption).

 

[D-FENS]

Exactly. Using a key is just like having a part of your password stored on disk. It's for additional flexibility, it does not add to the security. You could store all the password on a device, some of it on device and some of it in your head, or 100% of it in your head. Otherwise, the encryption does not care, it just needs the whole password.

 

[PMc]

For autodetection at boot time you need to set the geli flags "g" and "b" for an encrypted boot+root partition. I use a fully encrypted disk where only the tiny unencrypted bootloader is stored without geli.
By setting "b" and "g" you will be asked for password at boot time and geli will try to mount the partition as an encrypted root.

Thank You - I have seen these flags, but did think they do not apply to me. Because, what I expected to happen is not autodetection at boot, but autodetection when going multiuser.

Excourse:
It seems there are now three fashions to go for:

1. encrypt only selected, "private" drives.
2. encrypt the whole disk, but not the /boot subdirectory.
3. encrypt everything (the Allan Jude project).

From what I understood, the -b and -g flags seem to address case 2 and 3, while the fabric in /etc/rc.d is designed for case 1. In case 1, we do not need autodetection, because the user may attach and detach these special drives at their discretion.

But what seemed the most feasible solution to me is: encrypt everything except the OS. That means, the filesystems [/ /usr /var] stay unencrypted. (They also stay outside of ZFS on my installations.)
The big advantage with this is: if something goes wrong, one still has a fully functional single-user OS with all the tools to analyze the issue. In case 2 and 3, if the boot doesn't work, one has practically nothing - one can only boot from CD and hope to then find the issue. Why undergo that risk? There is nothing secret in the OS - one only needs to take care that none of the sensitive payload ever touches the /var subtree, i.e. those directories that are used by applications must be moved to separate filesystems.
That approach also leaves open the option to just export all the ZFS, and immediately have a system where I can test some kernel hacks or malfunctioning hardware or whatever.

So,thank You for the input - I will have a try with these options and see what they precisely do and if I can utilize that.

P.S. I use GPT. I am not aware what's the procedure for MBR.

If one wants the boot loader to access encrypted drives, I suppose one needs GPT. There is no space for that in MBR.

 

[D-FENS]

Sure. It all boils down to what exact scenarios do you want to protect against via the encryption.
If you want to protect only your documents and you are not concerned about somebody messing around with your system files, then leave the / unencrypted. It's also nice if you want to use the system remotely and be able to reboot it.
If you are worried that somebody might put malicious files on your / or /boot, then a full disk encryption is probably a good idea. However, without UEFI secure boot it's actually always possible to hack into your bootloader and install a keylogger or some other malicious program. It's not easy, but it's possible.

 

[PMc]

For autodetection at boot time you need to set the geli flags "g" and "b" for an encrypted boot+root partition. I use a fully encrypted disk where only the tiny unencrypted bootloader is stored without geli.
By setting "b" and "g" you will be asked for password at boot time and geli will try to mount the partition as an encrypted root.

That -b option does exactly what I was hoping for - that solves the issues! Great, thank You!!

Sure. It all boils down to what exact scenarios do you want to protect against via the encryption.
If you want to protect only your documents and you are not concerned about somebody messing around with your system files, then leave the / unencrypted.

This is what made me worry occasionally: I no longer exactly know what kind of stuff has been left as copies on the system, and which of it should be protected. Now say, a disk dies - what to do with it? I can destroy a spinning disk, but what about SSD? Burn it? And if it breaks while under warranty, the manufacturer wants it sent back.

So, I think, getting the user data fully encrypted answers a lot of these questions once and for all.
Getting it tamper-proof is a different matter, and doing that properly might get quite a bit more expensive...

It's also nice if you want to use the system remotely and be able to reboot it.

Ah, thats also an interesting aspect - one could get network up already in single-user.

Actually, the one big downside I did recognize in the password encrypted disk scheme is that the system will no longer come up on its own after a power loss (or crash, but ideally these should be fixed). So, one needs either an operator on-site, or a Diesel , or one of these fancy maintenance access features. None of these are simple solutions.

But actually there might be a simpler solution - I didn't think of that, because my remote access normally is a bit complicated (it involves a nested VPN) - but indeed, this is something I can ponder upon further.

If you are worried that somebody might put malicious files on your / or /boot, then a full disk encryption is probably a good idea. However, without UEFI secure boot it's actually always possible to hack into your bootloader and install a keylogger or some other malicious program. It's not easy, but it's possible.

There are more problems, depending on which powers we are up against...

 

[D-FENS]

There are more problems, depending on which powers we are up against...

Let's assume it's a grumpy colleague who wants to play a trick on you.

 

[linux->bsd]

This would present a chicken and egg problem. To access the encrypted data you need the key, that key is stored inside the encrypted data which can only be unlocked with the key, that's stored inside the encrypted data.

The chicken-egg dilemma would only arise if you were trying to use a keyfile for the zroot dataset with full-disk encryption. I've used the old unencrypted UFS boot partition setup with keyfiles stored on /boot, but now use full-disk encryption without a keyfile. It sounded to me like the OP was looking for a way to use keyfiles for his non-zroot dataset / geli device. So: you're prompted for the password for the geli device that contains zroot, proceed to the boot menu, then during the boot process the keyfiles are loaded for the non-root datasets, and away you go. But I didn't realize the OP was merely unaware of the "b" and "g" flags.

I've not tried that approach -- simply theorizing it should be possible based on when I'm prompted for passwords for non-root geli devices on my current setup. I think my root zpool is imported prior to the secondary password prompts, but could be mistaken.

 

[PMc]

It sounded to me like the OP was looking for a way to use keyfiles for his non-zroot dataset / geli device. So: you're prompted for the password for the geli device that contains zroot, proceed to the boot menu, then during the boot process the keyfiles are loaded for the non-root datasets, and away you go. But I didn't realize the OP was merely unaware of the "b" and "g" flags.

I underestimated the -b flag (I expected it to only look at the device for the root filesystem, while in fact it seems geli partitions are searched for on all drives) and overestimated the code in /etc/rc.d to do this thing.
But indeed this design makes sense: a zroot pool could be the only pool, and it could span many disks and partitions, and geli does not know the zfs layout, so it needs to just try to attach all it can find.

 

[D-FENS]

I underestimated the -b flag (I expected it to only look at the device for the root filesystem, while in fact it seems geli partitions are searched for on all drives) and overestimated the code in /etc/rc.d to do this thing.
But indeed this design makes sense: a zroot pool could be the only pool, and it could span many disks and partitions, and geli does not know the zfs layout, so it needs to just try to attach all it can find.

The b and g flags are applied to a specific partition. At boot time the loader searches through all devices and partitions and automatically tries to decrypt only the partitions with those flags set.
The flags do not have a global effect on all partitions!

 

[PMc]

The b and g flags are applied to a specific partition.

Yes, that is the way how one designates the drives belonging to the root pool (which shall be attached at boot time) against whatever-else. And this can also be used to attach a bunch of other pools, as long as their drives have the same passphrase.

 

[D-FENS]

You can in fact have different passwords for each encrypted partition.
If they all have the same password, you have to enter it only once. But if they have different passwords, you will be asked multiple times and you can enter all passwords.