Labelling geli disk with ZFS

xy16644

Aspiring Daemon

Reaction score: 13
Messages: 907

I've been using encrypted ZFS root for almost two years now and it has worked great on my old server. On my new server I'll be using pretty much the same setup/configuration but I was wondering: how do I stop my device names changing on my new server? This has never been an issue on my old server as the hardware setup was static but on my new server I'll be adding more disks over the coming years.

I think an example will explain the problem I have:

I have 8 SAS ports on my motherboard which are labelled SAS0 through to SAS7 on the ports itself on the motherboard. Currently my hard drives are connected as follows and have the following FreeBSD device names:

  1. First Samsung 840 Pro SSD connected to SAS0 port and is called da0 in FreeBSD
  2. Second Samsung 840 Pro SSD connected to SAS1 port and is called da1 in FreeBSD
  3. Samsung 1TB SATA drive connected to SAS7 and is called da2 in FreeBSD
FYI: The SSD drives are mirrored using ZFS.

This all works great if I don't ever change the disks. Now picture this: one of my SSD drives fails and I need to shutdown the server to remove it. When I power back up the 1TB Samsung SATA drive device name changes from da2 to da0 or da1! So the question I have is: how do I prevent my device names from changing at all? Ideally what I would like to do is reserve my SAS ports (0 through to 7) as follows:

SAS0 = da0
SAS1 = da1
...
SAS7 = da7

I have been reading about the following options so far but am unsure of how to proceed or which option to choose:

  1. Wire down the port. I have read about this online but there is little info regarding this and I am confused (or unsure) how to implement this. I did read it in the book Absolute FreeBSD but it still doesn't make sense to me.
  2. Someone mentioned to me about setting the following option in a custom kernel:
    ATA_STATIC in /usr/src/sys/i386/conf/GENERIC

    I don't have access to the server right now so I can't check if this option is set in GENERIC now on FreeBSD 9.2 STABLE. I think it is but my device names are still changing. This option is set on my old server in my custom kernel but I never enabled this myself so I assume its the default.
  3. Last but not least: labels. From what I understand you can't use glabels with geli encryption so this isn't an option. What I have heard from someone is that I can use GPT labels with geli; is this an option? If this is an option how can I start using GPT labels on an already built and configured encrypted ZFS root system? How do glabels differ from GPT labels?
Sorry for the long post but I'm just trying to understand what the best way to proceed is! I don't want my device names jumping all over the place when I add new disks, which can (and will) cause my system to become unbootable and cause encrypted partitions to not be accessible as the encryption keys can't be found etc.

Thanks!
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,744
Messages: 39,332

I've used various ways, they all essentially work the same way. On my latest drive I've used gpart(8) to create a GPT layout and one labeled freebsd-ufs partition. It shows up in /dev/gpt/. I then used /dev/gpt/mydisk.eli for ZFS. That way it wouldn't matter what device it gets assigned to. I haven't tried booting from such an encrypted disk though. With encryption I use it mainly on USB disks where the device name constantly changes. But for other systems I've used GPT labels and ZFS to boot from and it works really well. No more worrying about ada0 suddenly becoming ada4 when adding a controller :)
 
OP
X

xy16644

Aspiring Daemon

Reaction score: 13
Messages: 907

SirDice said:
I've used various ways, they all essentially work the same way. On my latest drive I've used gpart(8) to create a GPT layout and one labeled freebsd-ufs partition. It shows up in /dev/gpt/. I then used /dev/gpt/mydisk.eli for ZFS. That way it wouldn't matter what device it gets assigned to. I haven't tried booting from such an encrypted disk though. With encryption I use it mainly on USB disks where the device name constantly changes. But for other systems I've used GPT labels and ZFS to boot from and it works really well. No more worrying about ada0 suddenly becoming ada4 when adding a controller :)

So you *CAN* use GPT labels with geli and ZFS? I was always told (and under the impression) that labels wouldn't/couldn't work with geli.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,744
Messages: 39,332

Use them yes, I'm just not sure if you can boot from them. But for my USB disks it works well enough. Never tested its performance though.
 
OP
X

xy16644

Aspiring Daemon

Reaction score: 13
Messages: 907

I boot from a non-encrypted USB flashdrive with labels (with ZFS) but the root ZFS pool is entirely encrypted. Will labels work in this situation?
 
OP
X

xy16644

Aspiring Daemon

Reaction score: 13
Messages: 907

An update:

I created a new encrypted ZFS pool and I used labels with the GPT disk. I ran the following:
Code:
gpart create -s gpt da2
gpart add -t freebsd-zfs -a 4k -b 1M -l zstore da2
geli init -b -B /boot/zstore.eli -e AES-XTS -K /boot/encryption.key -l 256 -s 4096 /dev/gpt/zstore
geli attach -k /boot/encryption.key /dev/gpt/zstore
zpool create zstore /dev/gpt/zstore.eli

This created the ZFS pool and worked great:
Code:
       NAME              STATE     READ WRITE CKSUM
        zstore            ONLINE       0     0     0
          gpt/zstore.eli  ONLINE       0     0     0

I then added the following to /boot/loader.conf:
Code:
geli_da2p1_keyfile0_load="YES"
geli_da2p1_keyfile0_type="gpt/zstore:geli_keyfile0"
geli_da2p1_keyfile0_name="/boot/encryption.key"

When I rebooted it prompted me for the passphrase for da2p1. It did not accept my passphrase. After trying to enter my passphrase for 3 times it failed and then said: Enter passphrase for gpt/zstore. I entered my passphrase and it worked!

So the question I have is, what is the correct entries I need to put in the /boot/loader.conf file to be able to decrypt this disk which has a GPT label? I have tried the following unsuccessfully:

Code:
geli_da2p1_keyfile0_type="gpt/zstore:geli_keyfile0"
geli_da2p1_keyfile0_type="zstore:geli_keyfile0"
geli_da2p1_keyfile0_type="dev/gpt/zstore:geli_keyfile0"

I obviously don't want to have to enter the passphrase incorrectly 3 times before it will accept my passphrase.
 
Top