Other GEOM labels disappear after GELI attaches to raw disk device

malavon

malavon

I'm currently - well, for the past 3 weeks or so - in the process of recovering from a serious network intrusion at my home and am changing up most of my systems. Also, if oddities have shown up from my account then my account was compromised - just like every other account that I use, please private message me about that. Sadly I used to use a variation on the same password for every account and some have been leaked making it extremely easy for these hackers to compromise everything, included but not limited to deleting a dead friend's last facebook posts and several of my mails.
Anyway, right now I'm in the process of beefing up my internal security, which was also compromised. Each and every system was compromised because they managed to get a hold of my root password and combined with physical access, well, you know the drill. No way to stop that the way my systems were configured, i.e. assuming that the physical side of things would be safe.

So I use GELI* to encrypt partitions, asking for a passphrase at boot. I've been doing this for about a year on a laptop with a single NVME, but have now done the same on other systems. On systems with a few hard drives or more I used to create ZFS pools referencing GPT labels, but now that I switched to encrypted partitions I no longer do this because GELI and glabel would - as far as I know - write to the same sectors.

I decided to use the /dev/diskid/DISK-SNXYZ... references (from GEOM_LABEL in the kernel, kern.geom.label.disk_ident.enable=1 by default)., That's proven challenging.
At boot, the loader/GELI picks up the devices that should be decrypted and asks for a passphrase, but the devices picked up are the "raw" devices, e.g. /dev/ada0p1 etc. Once GELI has the right passphrase, the DISKID a.o. labels disappear. So, from that point on there is no more /dev/diskid/DISK-SNXYZ which I could use to import my zpools from.
For the record, these labels do exist after running geli detach ..., so they're clearly not disabled.

I can see a few options, I've been searching for them but might have searched for the wrong terminology:
1. force GELI somehow to use /dev/diskid/DISK-SNXYZ references
2. force GELI/GEOM to NOT hide the labels after attaching the encrypted device; hoping that the labels will appear encrypted as well
3. any other option that could help me reach my goal

If anyone knows of a way to help me reach my goal, please chime in. It's ok if this thread blows up with countless solutions, if I can find one suitable one it's perfectly worth it. Try not to heckle me though, I'm quite well aware of the risks I've taken and that I should have been much more paranoid about who I consort with :)

Thank you all!

* If you're wondering about why I use GELI instead of ZFS encryption, I prefer a FreeBSD-specific solution. Without having verified this statement, I'm assuming that these Kali Linux thumbdrive "hackers" will have a bad day decrypting my drives and need to resort to a system they know much less about. Let's just call it evening the playing fields, or much rather, tipping the favor my side.
 
Commiseration for the intrusions.

My first thought is to suggest that you don't over-react.

You clearly had some poor security choices. BTDT, but now reformed.

Sit back and consider the risk analysis. What are you really trying to protect, what are the genuine risks, and what are the potential regrets.

geli(8) can offer very strong protection, especially against loss or theft of physical media. But you might be surprised how many people manage to lose or corrupt their geli(8) key(s) and thus lose everything! A failure with geli(8) has the potential to be of huge regret. So geli(8) is actually a major risk, in itself.

Besides which, if you have a physical intruder with root access, geli(8) may not help!

I have worked in some well-secured facilities. We never used whole disk encryption (but all used disks were sent for destruction in a chipper). Good design, strong passwords (kept in a safe), and well managed ssh(1) keys can go a very long way to affording sound protection.

geom(8) providers are required to stack, indefinitely. Order matters because each class has the option to place metadata in a provider's last sector. These metadata are private, and the space they occupy must not be offered for use higher in the stack. So, as you configure classes progressively up the stack, you must choose the device name appropriate for the provider to that class with great care.

glabel(8) is a geom(8) class. In "automatic" (permanent label) mode it keeps the label in a provider's last sector. The devices to use for the glabel(8) provider are in /dev/label.

Aside: in a quirk, labels inside partitions are kept with the rest of the partition metadata (they don't need to reserve space in the last sector). geom(8) understands this anomaly.

Have you tried something like:
Code: 
glabel label -v disk-SN0 /dev/ada0
geli init /dev/label/disk-SN0
You may have to ensure that geli(8) and glabel(8) kernel modules are loaded at boot.

Edit: I think that glabel(8) probably only works on whole disks when the MBR partitioning is used, because both glabel(8) and gpart(8) want to put metadata in the last sector (and gpart(8) does not play by the rules of geom(8)).
 
Thanks @gpw28 for raising some questions and extra considerations.

First the hack: trust me, I'm not overreacting. It's not even done yet, my cable modem has been hacked and I have to be very careful about what I do. fyi I'm writing this from a different connection. I expect these people to get back to my systems as soon as I leave the house for an extended period. They need to be prepared adequately for that occasion.
Also, packages on my pc were modified to send passwords back to my hackers, I have to assume they have all my passwords no matter how careful I am or have been. The geli keys however have been set offline and are used offline - there's no network connection configured at the time of boot.

Second geli: correct, if I lose the passphrase I'm screwed. That is true for all disk encryptions. I don't use keyfiles, so it isn't an issue to lose these. Not sure if that's as secure as I'm hoping though. That said, GELI metadata can be backed up of course.

The labels I'm looking for aren't GPT labels as set by glabel, in fact I used to use these but no longer do. I now want to use the labels that are automatically provided by geom_label (GEOM_LABEL in kernconf) and put under /dev/diskid/. These are simple automatically-generated labels based on the disks brand and serial number as far as I can see. For servers with e.g. 12 disks it will still be useful. I used to use GPT labels (without GELI) like "wd_red_4tb_bay_1", so using serial numbers is going back a bit but I'm fine with that.

@gpw28, what you're showing with glabel label ... is setting a custom GPT label (not also: they don't have to be set on disk, can be used on a partition). I now have this disabled specifically to address any issues between them. No matter though, even if I did use GPT labels they would disappear as soon as GELI mounts the raw device at boot. Ergo, back to the start I'm afraid. I have tried before using geli init on one of these labels, but it doesn't change the way GELI picks up the devices at boot.
 
malavon said:
I decided to use the /dev/diskid/DISK-SNXYZ... references (from GEOM_LABEL in the kernel, kern.geom.label.disk_ident.enable=1 by default)., That's proven challenging.
Click to expand...
specifically for this, sysctl control this:
kern.geom.label.disk_ident.enable
It defaults to enabled, but is controlled by "geom withering" as do all the others like gptid. I would double check how things are mounted under the GELI bits.
 
mer said:
but is controlled by "geom withering" as do all the others like gptid. I would double check how things are mounted under the GELI bits.
Click to expand...
I did notice that withering code in the sources, wondering what it was. From g_wither_geom(9) I don't think that's what hides the labels, but thanks to your search term I ended up on this archived mail thread where it is explained to be completely normal. Apparently the test I've done before must have been prejudiced, I probably mounted the "/dev/diskid/DISKID-SNXYZ", which obviously remained after mounting.
In short: upon mounting a disk - through any label or raw device - the provider is consumed and all unused labels disappear, ergo this is intended behaviour. I'll have to look elsewhere than the GELI/GEOM sources.

So, I guess that the question now singularly becomes: can I force GELI (or the loader) to only look at /dev/diskid/ entries instead of the raw ones? From the loader.conf options in geli(8) I'd say not, but more investigation needs to happen now. I'll be looking at the loader sources next to see if there's anything pertaining to GELI/GEOM.
 
In my loader.conf I explicitly disable diskid and gptid labels so I would guess that the loader and other GEOM layers would see everything until something uses one of the labels. I've not done anything with GELI so I'm going by documentation here so double check everything.

Anyway, my understanding is that with GELI you associate a key/keys with a partition, so if you use /dev/diskid/whatever when you create the GELI stuff you should still see that form of the device.
If you use /dev/ada0p1 (or whatever) when you create the GELI stuff you would lose the diskid labels.
 
Some more stuff: fusefs (or at least ntfs-3g) doesn't seem to be hiding the (/dev/diskid/) labels. zpool import does when mounting from the raw disk device(s), but zpool export doesn't restore/release the labels afterwards. Tested on a USB-mounted pool which made it easy to reset to original state. No labels disappear when originally zpool import from a label. Meaning: when imported from /dev/diskid/XYZ, the /dev/gptid/01234 labels also remain.
I retract - or better yet - correct the statement I made in previous post: upon mounting a (GEOM or regular) disk - through the raw device - all labels disappear. They are restored when unmounting, except when part of a ZFS pool.
Question in last post thus still stands :)
 
mer said:
In my loader.conf I explicitly disable diskid and gptid labels so I would guess that the loader and other GEOM layers would see everything until something uses one of the labels. I've not done anything with GELI so I'm going by documentation here so double check everything.
Click to expand...
I used to do this too, I only used gpt labels and didn't want to be confused by the others. But now I want to switch to only diskid :)
mer said:
Anyway, my understanding is that with GELI you associate a key/keys with a partition, so if you use /dev/diskid/whatever when you create the GELI stuff you should still see that form of the device.
If you use /dev/ada0p1 (or whatever) when you create the GELI stuff you would lose the diskid labels.
Click to expand...
This is what I expected too, but apparently that's not the case. I made several GELIs on top of these /dev/diskid/ labels; and created zpools on them, but as soon as the loader came up it would only present the raw devices. Hence why one of my solutions was to keep the labels despite being already "consumed" by GEOM/GELI. Or alternatively why I was hoping to force the devices to be taken from /dev/diskid instead.
The only possibility maybe - but that's a definite maybe - is that I may not have had the /dev/diskid/ labels enabled when I first booted from the GELI, but I'm almost 100% positive that one the 5 systems where I did this I did it right at least once (because it's a fresh install from original FreeBSD image). Might be worth checking though, I should be able to test this on at least one system. Will do so now.
 
  • Like
Reactions: mer
This may be interesting, it's a couple years old but the fundamentals should be the same and sounded exactly like what you are looking for.

Rubenerd

Enable the appropriate GEOM lines in loader.conf
rubenerd.com rubenerd.com
 
And test has concluded:
1. removed raw (GELI) device from a (mirror) ZFS pool
2. geli init ... /dev/diskid/DISK-SNXYZp1
3. geli attach /dev/diskid/DISK-SNXYZp1
4. zpool attach s /dev/<mirror-dev> /dev/diskid/DISK-SNXYZp1.eli
5. Waited for resilver to be done
6. Reboot
7. GELI attached at boot with raw devices, /dev/diskid/ labels disappear at boot time
8. ZFS import picks up /dev/<rawdev>.eli only since there is no /dev/diskid label anymore

So in short: geli init from /dev/diskid does not solve the issue, provably.
 
The article I linked above talked about explicitly enabling the disk_ident sysctl in loader.conf; perhaps that's needed?
 
mer said:
This may be interesting, it's a couple years old but the fundamentals should be the same and sounded exactly like what you are looking for.

Rubenerd

Enable the appropriate GEOM lines in loader.conf
rubenerd.com rubenerd.com
Click to expand...
I'm afraid that that post doesn't tell me anything new. I'm quite well aware of GEOM labels, it is this particular issue that I have problems with.
I'm also quite certain that these diskid labels would have been enabled by default in 12.1 as that article seems to be mentioning so I wouldn't put too much faith in that blog.
Nonetheless I tested with "kern.geom.label.disk_ident.enable=1" in /boot/loader.conf. Predictably the result is the same, GELI attaches only to the raw disk devices and hides the labels afterwards.
 
  • Like
Reactions: mer
Crivens said:
Physical access goes both ways...
Click to expand...
Feel free to explain your statement, I fail to see what it means. If it means I would be there while my systems are physically under assault, it foregoes the premise that it happens while I'm away. If it means I would have access to their systems too I again fail to see how, since I won't magically be provided a key to their establishment to go and mess up their systems. /joke
 
Crivens said:
Easy. You should know who did this, do you?
Someone doing this to me and knowing I know would not expect nothing to happen.
Click to expand...
Oh, don't worry about that... 👿 ...but there's much more to deal with right now aside from my systems. Let's just call it a perfect storm, albeit engineerd to coincide I'm afraid. Anyway, preventing more damage is my first priority right now and I'm quite occupied with that. Frankly, the questions in this thread aren't even high on the priority, but it's all about having usable systems afterwards. I wouldn't want to be stuck with raw disk devices if I can help it.
 
malavon said:
I decided to use the /dev/diskid/DISK-SNXYZ... references (from GEOM_LABEL in the kernel, kern.geom.label.disk_ident.enable=1 by default)., That's proven challenging.
At boot, the loader/GELI picks up the devices that should be decrypted and asks for a passphrase, but the devices picked up are the "raw" devices, e.g. /dev/ada0p1 etc. Once GELI has the right passphrase, the DISKID a.o. labels disappear. So, from that point on there is no more /dev/diskid/DISK-SNXYZ which I could use to import my zpools from.
Click to expand...
List the geli(8) provider to automatically attach (after entering the passphrase) at boot in /etc/rc.conf in the geli_devices variable (see /etc/defaults/rc.conf for further geli(8) variables). From there, the geli(8) providers are attached at a later booting stage, after the diskid's are created.

I've tested the following setup in a VirtualBox VM (13.2-RELEASE):

A two disk system (ada0, ada1). ada0 is the system disk, ada1 for the encrypted ZFS pool.

/boot/loader.conf
Code: 
kern.geom.label.disk_ident.enable="1"

Code: 
 # gpart create -s gpt ada1
 # gpart add -t freebsd-zfs -a 1m -l zpool-tank ada1

 # gpart show -lp
=>    40  8388528    ada1  GPT    (4.0G)
      40     2008          - free - (1.0M)
    2048  8384512  ada1p1  zpool-tank  (4.0G)
  8386560    2008          - free - (1.0M)

=>     40  8388528    diskid/DISK-VB6630C644-6873fb47  GPT  (4.0G)
       40     2008                                     - free - (1.0M)
     2048  8384512  diskid/DISK-VB6630C644-6873fb47p1  zpool-tank  (4.0G)
  8386560     2008                                     - free - (1.0M)

Code: 
 # geli init -l 256 -s 4096 diskid/DISK-VB6630C644-6873fb47p1
 # geli attach diskid/DISK-VB6630C644-6873fb47p1

 # zpool create tank diskid/DISK-VB6630C644-6873fb47p1.eli

/etc/rc.conf
Code: 
geli_devices="diskid/DISK-VB6630C644-6873fb47p1"

Reboot system, enter mid-boot geli(8) passphrase when prompted.

Code: 
 # geli status
                                 Name  Status  Components
diskid/DISK-VB6630C644-6873fb47p1.eli  ACTIVE  diskid/DISK-VB6630C644-6873fb47p1

 # zpool status tank

  pool:  tank
 state:  ONLINE
config:

         NAME                                     STATE     READ WRITE CKSUM
         tank                                     ONLINE       0     0     0
           diskid/DISK-VB6630C644-6873fb47p1.eli  ONLINE       0     0     0
    
errors:  No known data errors
 
T-Daemon said:
A two disk system (ada0, ada1). ada0 is the system disk, ada1 for the encrypted ZFS pool.
Click to expand...
I almost loved your suggestion because I thought you were talking about an unknown or hidden loader variable :cool: I thank you a thousand times for testing it out, I really do, but alas I can't, or more precisely, don't want to use a non-encrypted (system) disk. All disks need to be encrypted at boot, that's really a hard requirement for me. If that means using raw devices then so be it.
The whole idea is that it's impossible - or at least incredibly hard - to tamper with my system by booting from a USB stick, from lan etc. As soon as there is a possibility for an attacker with physical access to install software, modify configuration files or anything else that compromises my systems from within then it's a non-viable solution. This is also the reason why I'm experimenting a lot with pretty much all security that FreeBSD has to offer, starting from kern.securelevel etc (which appears not to be possible on my desktops with GPU drivers - but that's for another thread) combined with filesystem flags & ZFS settings.
T-Daemon said:
From there, the geli(8) providers are attached at a later booting stage, after the diskid's are created.
Click to expand...
Assuming the loader has no knowledge about these labels - which would be logical since the kernel hasn't loaded yet to create the geom labels - while geli attaches, I'm guessing the only option would be to limit the kernels ability/willingness to remove the GEOM labels after GELI attaches a device. Do you, or anyone else reading this, know where in the sources this might be happening? I don't mind building from modified sources - I already build my custom world & kernel now anyway - if that means I can use diskids for my encrypted ZFS pools.
 
Following up myself: I might be able to do something with GEOM multipath, it's like a label but I'd have to recreate all pools. Not impossible, just not optimal.

Also, the disappearing of devices would appear to be - rather logically - called orphanization. But this orphanization can not happen when a provider is attached. So, theoretically, if I could attach a ZFS pool on the geli version of the /dev/diskid/DISK-XYZ then hypothetically it would attach exclusively, thus the label would not disappear afterwards. And I can make sure the zpool uses these labels, but now I have remember - I seem to have something buried deep in my memory - where I read about this.
 
Fyi, just to be complete and in case anyone reading this thread doesn't know this: the FreeBSD installer creates bootable (root!) GELI volumes which are decrypted by the - in my case EFI - loader. That means that aside from an EFI partition containing the bootloader, there's nothing unencrypted on the drives. If you want to create one of these yourself you can simply add the -g flag to geli init. The installer also adds -b but I think that's not necessary. That one is used by the GELI rc.d script to find geli devices that are eligible to be attached during the rc.d phase of the boot. This is one simple way to omit the "geli_devices" in rc.conf if you're mounting GELI devices during the later stages of the boot.

That said, back to the problem at hand. If there is a way to prevent the labels from disappearing for long enough that the ZFS pools have time to be imported - after which ZFS will make them disappear I reckon - then I should be able to do everything I want as far as GELI goes. Because once that is tackled, it's rather easy to swap every pool that I have into any type of label by booting on a FreeBSD bootable USB using zfs import -d /dev/diskid/ <pool> followed by a zfs export <pool>.
 
Last edited:
You must log in or register to reply here.
Back
Top