HOWTO: FreeBSD 10.1 amd64 UEFI boot with encrypted ZFS root using GELI

asteriskRoss

asteriskRoss

Introduction
In this HOWTO, we'll walk through installing FreeBSD 10.1-RELEASE as the sole operating system on a UEFI-enabled amd64/x86-64 PC to a single hard disk, with all except the /boot directory installed to a ZFS pool encrypted using geli(8).

The /boot directory will reside on an unencrypted UFS partition.

If you're reading this for a later version of FreeBSD then it will probably work, but there may be a better and easier way of achieving the same goal. FreeBSD 10.1 was the latest release at time of writing.

Understanding technical limitations
UEFI support was added to FreeBSD in the 10.1 release for the amd64 architecture, but has some limitations:
  • It only supports booting from a UFS partition (that is, not ZFS)
  • It does not support UEFI Secure Boot.
PC manufacturers' implementations of UEFI vary in quality. During the transition from legacy BIOS booting to UEFI, many manufacturers include a method of booting from both. This might be configurable or the firmware may decide which one to use based on the disk partitioning type (MBR or GPT) or presence of boot sector code. Even if your PC supports UEFI, implementation issues may prevent this method working for you.

The configuration described here is not compatible with the ZFS Boot Environment management utilities sysutils/beadm or manageBE, since both of these make assumptions about the filesystem layout that aren't true here.

If you're using an SSD, you should know that geli(8), which we are using here for encryption, doesn't yet support TRIM, which will unfortunately have implications for your write performance.

A brief discussion on risk mitigation, disk encryption and GELI
This is not a HOWTO on different disk encryption techniques but you should understand what protection this configuration offers and what it doesn't. When designing security, it is important to keep in mind whom you are defending against. In this configuration, I'm aiming to prevent someone reading my data if I lose the computer (all too common for laptops) or if it is stolen by a thief more interested in selling the hardware for cash than for any secrets on the hard disk. I am not looking to protect my data from espionage level attacks or from covert modification.

Encrypting information on a disk protects an attacker from accessing it "at rest", that is, when the computer is powered off. It offers no protection at all against attacks while the computer is powered on and you have made that information available in its decrypted (plain text) form. This is true for all disk encryption. The configuration described here has further shortcomings. Secure Boot is disabled, the kernel and its modules are available in unencrypted form on the disk and I will be using GELI without enabling data authentication. This means that if someone sneaky wants to plant attack software on the machine, conduct an "evil maid" style attack or even modify the encrypted data so it decrypts to something different, they can and you won't know about it.

If I had different requirements, I would consider putting my UEFI boot files and kernel on a removable disk that I kept with me, enabling data integrity verification for my GELI partition, encrypting with AES 256-bit keys rather than 128-bit, physically securing my PC and making it tamper evident, locking down the firmware configuration, rewriting the UEFI bootloader to support Secure Boot, using a PC with a TPM chip, reviewing the FreeBSD source code, never connecting my computer to the Internet, installing an alarm system in my office, training an attack dog to guard my computer... you get the idea. You can hire me for security consultancy or attack dog training at competitive rates but for now, let's get on with the show.
 
Preparation
We're going to be overwriting whatever is currently on your hard disk so make sure that you have backed everything up! If this is one of your first few times installing FreeBSD, make sure you're familiar with the installation chapter in the Handbook and the preparation suggested there.

We'll also need installation media from your nearest mirror. I suggest grabbing the UEFI memory stick image FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img, since this will allow you to tell early on whether UEFI booting will work fine for you. Write the image to a memory stick (see section 2.3.1.1 in the Handbook).

Check your PC or motherboard manufacturer for any UEFI firmware updates, which may include bug fixes.

Configuring your PC firmware boot settings
You want to use UEFI, right? Power on your PC and press whatever key is required to enter your BIOS settings (Esc, F2 and Insert are common possible options). Refer to any documentation from your PC or motherboard manufacturer if you need to. Have a look through the menu and try to identify and configure settings to:
  • Enable UEFI boot
  • Disable Secure boot
  • Disable Legacy boot, which may be labelled MBR boot or CSM module
  • Disable Fast Boot (relating to Windows).
Save your changes. As mentioned earlier, your manufacturer's UEFI implementation may not allow you to configure these.

Confirming your PC can boot to UEFI
Plug the FreeBSD UEFI installation memory stick into your PC. When the PC powers up, you may need to press a key to choose a boot device (Esc, F9 and F12 are common possible options). Hopefully you will see a menu, which includes an entry for the memory stick, or perhaps just a generic "UEFI USB Drive". Select this to boot to the FreeBSD installation media.

If you have booted successfully using UEFI, you should see a high resolution console with the first line ">> FreeBSD EFI boot block". Hit enter to boot immediately or wait for the counter to reach zero. After the initialisation process completes, you should land in the blue installation menu. Congratulations! Your PC should be fine with the FreeBSD UEFI loader.

If your kernel panics on boot, then UEFI is not for you. If the FreeBSD USB stick wasn't listed in the boot options menu, try checking your firmware boot settings again. Just to confuse you (and possibly your PC's firmware), the installation memory stick is actually a hybrid that supports UEFI and legacy BIOS booting. If you are not able to explicitly disable legacy boot in your PC's firmware, it is possible that you have instead booted using legacy BIOS. If so, you will see the first line "BTX Loader" before arriving at the FreeBSD boot menu with ASCII art. Although this isn't a successful result, it does not mean all is lost. Our installation will be pure UEFI, not a hybrid so you may choose to carry on and hope that all comes good.
 
Starting the installation
At the welcome screen, choose Install and then choose your keymap, hostname and distribution sets as desired. When you reach the partitioning screen, choose "Shell Open a shell and partition by hand". Now, my young apprentice, the fun begins.

Partition your disk
We'll assume here that your PC has a single disk, which FreeBSD identifies as ada0. Adjust the commands for your configuration as required. We're going to create four partitions:
  1. EFI boot partition
  2. Swap partition (that we'll configure to be encrypted with geli(8) whilst in use)
  3. UFS partition containing /boot, which includes the FreeBSD kernel
  4. Partition encrypted with geli(8) as a container for our ZFS pool.
The EFI boot partition can be very small if you like (the single file you need is only 36KB) but here I've allowed 100MB to give plenty of room for other UEFI utilities you may want to install in the future. I've seen Microsoft Windows installations with 500MB EFI partitions. In theory, this can be formatted as FAT12, FAT16 or FAT32. I found that my PC's firmware wouldn't recognise a FAT32 partition created under FreeBSD and the UEFI wiki page says:
FreeBSD's FAT32 code appears to sometimes create filesystems that the UEFI code can't properly read.
Click to expand...
FAT16 worked fine and this is what we'll use here.

We need the UFS partition since for FreeBSD 10.1-RELEASE, that is the only partition type supported by the UEFI loader. A sensible minimum size for this partition should allow enough space for the boot files and three copies of the kernel and modules (running kernel, newly installed kernel and a copy of the GENERIC kernel), which is about 1.5GB. We'll go for 10GB to give us more space than you can wave a stick at.

I've chosen to create a dedicated swap partition. You may be tempted to put your swap space on a ZFS Volume on the ZFS pool inside the GELI container. I believe this is a bad idea because if it is ZFS that needs the extra memory to work properly, it may not be able to provide that swap space. How much swap space to allocate is up to you. Section 2.6.1 of the Handbook says:
As a rule of thumb, the swap partition should be about double the size of physical memory (RAM).
Click to expand...
Since this rule of thumb has existed from when 64MB was an enormous size of RAM, for modern systems that amount may be excessive. I've allowed 8GB swap here, which should be far more than you will ever need on a typical system.

We'll also be aligning our partitions to 1MB boundaries, which is never the wrong thing to do. You could save a small amount of disk space by not bothering at all with alignment or instead aligning to 4K boundaries. It's relevant for performance on Advanced Format Drives, SSDs and RAID volumes but comprehensive discussion is off-topic for this HOWTO.

You are free to disagree and do something different for any of this, but this is my HOWTO party and I'll cry if I want to... erm I mean design my partitioning how I want to. Please find somewhere other than this thread to argue.

Getting down to business: Creating the partitions
Destroy any existing disk partitioning (you did take a backup, right?) with:
gpart destroy -F /dev/ada0

Create a GUID Partition Table (GPT) scheme:
gpart create -s GPT /dev/ada0

Add the EFI partition of 100MB with the label "EFI" aligned to 1MB:
gpart add -t efi -s 100M -a 1M -l EFI /dev/ada0

Add the swap partition of 8GB with the label "FreeBSD-swap" aligned to 1MB (we don't actually need the alignment explicitly specified for additional partitions if they are all whole megabytes in size, but it doesn't hurt to be explicit):
gpart add -t freebsd-swap -s 8G -a 1M -l FreeBSD-swap /dev/ada0

Add the UFS partition of 10GB with the label "FreeBSD-ufsboot" aligned to 1MB:
gpart add -t freebsd-ufs -s 10G -a 1M -l FreeBSD-ufsboot /dev/ada0

Add the partition in which we'll configure our encrypted container for our ZFS pool, using all remaining space on the disk with the label "FreeBSD-enczroot" aligned to 1MB:
gpart add -t freebsd-zfs -a 1M -l FreeBSD-enczroot /dev/ada0
 
Configuring the EFI partition
Create the FAT16 filesystem for the EFI partition, with a label of FreeBSD_EFI:
newfs_msdos -F 16 -L FreeBSD_EFI /dev/ada0p1

Create a temporary mountpoint for the new FAT filesystem:
mkdir /tmp/efi

Mount it:
mount -t msdosfs /dev/ada0p1 /tmp/efi

Create the directory for the FreeBSD UEFI bootloader:
mkdir -p /tmp/efi/EFI/BOOT

Copy the FreeBSD UEFI first-stage bootloader into that directory with the name the firmware expects:
cp /boot/boot1.efi /tmp/efi/EFI/BOOT/BOOTX64.EFI

Unmount the filesystem:
umount /dev/ada0p1

If your UEFI boot test failed earlier, now is good time to try to boot to the hard disk and see whether a pure UEFI boot (rather than a hybrid) is going to work. We haven't installed the kernel yet, so don't expect the boot loader to find one!

Configuring the UFS partition
Create the UFS filesystem with a label of "ufsboot" and a sector size of 4KB. If you are installing to an SSD that supports TRIM, you should also add the -t flag:
newfs -L ufsboot -S 4096 /dev/ada0p3

Create a temporary mountpoint for the new UFS filesystem:
mkdir /tmp/ufsboot

Mount it, as we're going to create a GELI key file to store on it:
mount /dev/ada0p3 /tmp/ufsboot

Configuring the GELI container
We're going to configure our GELI container using the AES-XTS algorithm (recommended in the geli(8) man page, though AES-CBC would also be fine) with 128 bit key length for encryption and no(!) data integrity verification. If your processor implements AES-NI instructions, this will speed up encryption and decryption using AES-XTS or AES-CBC. Whilst data integrity verification is really very sensible, it means a performance hit as there is no hardware acceleration for the HMAC functions. 256 bit key length also means lower performance. With the caveats I offered earlier in this HOWTO, think about your security requirements, do some reading and decide what you want to do.

Given this pragmatic (some may say cavalier) approach to disk encryption, you may be surprised that I'm going to use a key file in addition to a passphrase. Why, you ask? Why not? There are no performance implications and it helps to mitigate an attack using a rainbow table, where someone could compute in advance all the possible hashes of the passphrase; the key file is effectively salt here. Let me caveat this statement by saying that I've not delved into the source code to establish whether GELI adds its own salt but since the man page isn't explicit and there is no drawback we'll go for it.

Load the AESNI kernel module. If you didn't know already the output from this will tell you whether your CPU supports AESNI:
kldload aesni

If your CPU does support it, you should see something like the following line on the console:
Code: 
aesni0: <AES-CBC,AES-XTS> on motherboard
Create a folder to store GELI related files on /boot:
mkdir -p /tmp/ufsboot/boot/geli

Create a key file filled with 64 bytes (512 bits) of random data:
dd if=/dev/random of=/tmp/ufsboot/boot/geli/ada0p4.key bs=64 count=1

Initialise the GELI device to encrypt with AES-XTS, 128 bit key length, 4KB sector size, mounting automatically on boot, before the root partition is mounted, with the keyfile we just created. Choose a strong passphrase:
geli init -e AES-XTS -l 128 -s 4096 -b -K /tmp/ufsboot/boot/geli/ada0p4.key /dev/ada0p4

Take a copy of the GELI metadata, which you will otherwise lose when you reboot:
cp /var/backups/ada0p4.eli /tmp/ufsboot/boot/geli/

Attach the GELI device, entering your passphrase when prompted:
geli attach -k /tmp/ufsboot/boot/geli/ada0p4.key /dev/ada0p4

Configuring the ZFS pool
We're now ready to create our ZFS pool inside our GELI container. We're not going to write files to the ZFS dataset with the same name as the pool, since unlike other datasets you can't destroy it later. Instead we're going to create a structure amenable to using boot environments, though noting that our configuration isn't compatible with the boot environment management tools and so all dataset manipulation will have to be done by hand. As performance tweaks, we're going to disable recording of access time to files when they are read and also enable compression. Compression and decompression are fast, while GELI encryption and decryption and reading from disk are slow. Using compression means less data to encrypt, less to decrypt and less to read and write from disk. It will improve performance but feel free to run some benchmarks to satisfy yourself.

Create a ZFS pool, with /mnt as the alternate mountpoint (where the installer will put our new FreeBSD system) called zroot with desired options for the zroot dataset, which will be inherited by its children:
zpool create -R /mnt -O canmount=off -O mountpoint=none -O atime=off -O compression=on zroot /dev/ada0p4.eli

Container for boot environments:
zfs create -o canmount=off -o mountpoint=none zroot/ROOT

Default boot environment:
zfs create -o mountpoint=/ zroot/ROOT/master

Things we want to be unique for each boot environment:
zfs create -o mountpoint=/usr/jails zroot/ROOT/master/jails
zfs create -o mountpoint=/usr/local zroot/ROOT/master/local
zfs create -o mountpoint=/usr/ports zroot/ROOT/master/ports
zfs create -o mountpoint=/var zroot/ROOT/master/var
zfs create -o mountpoint=/var/log zroot/ROOT/master/log

Things we want to be common across boot environments:
zfs create -o mountpoint=/usr/home zroot/home
zfs create -o mountpoint=/usr/obj zroot/obj
zfs create -o mountpoint=/usr/ports/distfiles zroot/distfiles
zfs create -o mountpoint=/usr/src zroot/src101
zfs create -o mountpoint=/tmp zroot/tmp
zfs create -o mountpoint=/var/tmp zroot/vartmp
 
Last edited:
Prepare for FreeBSD installation
We need to put all our filesystems in the appropriate place under /mnt for bsdinstall(1) to do its thing. Our zroot pool is already there.

Create a permanent mountpoint for our UFS filesystem:
mkdir /mnt/ufsboot

Remount our UFS filesystem there:
umount /dev/ada0p3
mount /dev/ada0p3 /mnt/ufsboot

So /boot appears in the expected place, we'll need a symbolic link:
( cd /mnt && ln -s ufsboot/boot boot )

Create fstab(5) for the UFS partition and our encrypted swap by creating /tmp/bsdinstall_etc/fstab:
vi /tmp/bsdinstall_etc/fstab

Fill it with the following to encrypt the swap space with the same encryption method as our GELI container and to mount our UFS partition on boot:
Code: 
# Device   Mountpoint   FStype   Options   Dump   Pass#
/dev/ada0p2.eli   none   swap   sw,ealgo=AES-XTS,keylen=128,sectorsize=4096   0   0
/dev/ada0p3   /ufsboot   ufs   rw   1   1
Exit back to the installer and complete the installation with:
exit

Choose the root password, configure your network interfaces, your timezone, services to be started at boot, add any users and exit to reach the "Manual Configuration" screen.

Final configuration
Don't reboot yet but choose Yes to open a shell as we need to make some configuration changes.

Edit loader.conf(5):
vi /boot/loader.conf

Fill it with the following content to configure the GELI container attachment, load the ZFS module and tell the system where to find the root filesystem:
Code: 
aesni_load="YES"
geom_eli_load="YES"
geli_ada0p4_keyfile0_load="YES"
geli_ada0p4_keyfile0_type="ada0p4:geli_keyfile0"
geli_ada0p4_keyfile0_name="/boot/geli/ada0p4.key"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot/ROOT/master"
Edit rc.conf(5):
vi /etc/rc.conf

Append the following line to mount all your ZFS datasets on boot:
Code: 
zfs_enable="YES"
exit back to the installer, select Reboot. Remove the memory stick and keep your fingers crossed!

Troubleshooting
If it doesn't work first time, don't despair! Boot to the install media, choose LiveCD and log in as root with a blank password. Mount the UFS filesystem and check over the configuration.

The kernel and modules load, but you're not prompted for a passphrase and the system is unable to find the root filesystem
A common error with GELI is forgetting to set the flag on the container so that GELI mounts it during boot. You can set the flag by:

Mounting the UFS filesystem (to get the GELI keyfile):
mkdir /tmp/ufsboot
mount /dev/ada0p3 /tmp/ufsboot

Attaching the GELI container:
geli attach -k /tmp/ufsboot/boot/geli/ada0p4.key /dev/ada0p4

Configuring the GELI container:
geli configure -b /dev/ada0p4.eli

I know I typed the correct pass phrase but GELI says it's wrong
You might be typing the wrong passphrase, but more likely you made a small mistake when typing the GELI key configuration in /boot/loader.conf. Check the path to the key file and that the keyfile exists.

Further reading
 
Last edited:
I just followed this guide myself, and got the system up and running. Thanks, asteriskRoss. However, I have a self-encrypting SSD (Samsung 840 EVO) and so have no need for geli(8). If anyone else is in my situation, or if anyone chooses to follow this guide without using encryption for some reason, a caveat must be noted. The FreeBSD EFI launcher progresses through four rapid stages:
  • The motherboard firmware searches for the EFI partition on the disk
    • The firmware then finds the EFI binary and executes it
      • The EFI binary then searches for the first UFS partition it can find
        • Once the first UFS partition is found, the EFI executable looks for the file /boot/loader.efi
If any links in that chain are broken, the EFI loader will panic. So if you skipped the step in this guide on creating a geli key, you may also have missed the creation of the boot directory nested inside the UFS boot partition. The EFI loader will not find the boot files if they're stored in the top level of the partition---hence the need for the symlink to /boot mentioned in the guide. Just thought I'd explicitly state that for posterity.
 
ANOKNUSA: I'm pleased you found the HOWTO useful even though you are not using GELI.

Regarding the need for a /boot directory, the way I think about it is that the boot loaders are written for a standard system, where the kernel and related files are in the /boot directory on the root partition. Even if the kernel and other boot files are moved elsewhere (as in this HOWTO), the boot loaders still expect that directory structure.
 
I've had good success (on OSX) by putting the contents of /boot/ directly into the EFI partition. I use refind as a boot manager, but this does work without refind/OSX installed at all. By excluding the symbols I can fit everything in the 200MB including a linux kernel as well - triple boot with OpenZFS on all 3 platforms :D

Code: 
# mount your EFI partition only to do a copy
# this way you have a known good version in zfs always
rsync -harv /boot/ /efi/boot --exclude=\*.symbols --del --partial --inplace

Code: 
# interesting snippets from /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.vty="vt"
# zfs
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot/ROOT/default"
 
Last edited by a moderator:
Bucky, I'm pleased you found the HOWTO useful. Regarding choice of encryption algorithm I agree that the geli(8) default of 128-bit AES-XTS (as configured by someone following the HOWTO instructions) is the one to choose. Before the performance improvements (see the reference from the "further reading" list; BSDCan2014: Optimizing GELI performance), the choice wasn't so easy as the AES-CBC implementation was significantly faster. With my stated aim in mind...
asteriskRoss said:
In this configuration, I'm aiming to prevent someone reading my data if I lose the computer (all too common for laptops) or if it is stolen by a thief more interested in selling the hardware for cash than for any secrets on the hard disk. I am not looking to protect my data from espionage level attacks or from covert modification.
Click to expand...
...I believe that either algorithm is good enough. As I alluded to in the brief discussion on risk mitigation if I were looking to protect my FreeBSD installation and information from a more serious threat I would take a different approach and accept the associated convenience and performance penalties. Depending on your requirements you may want to consider a different design for your system to the one suggested here. However, before you start using a removable boot device you keep on you at all times, 256-bit key length and data integrity verification I also refer you to my tongue-in-cheek reference to the xkcd cartoon linked in my "further reading" list:

security.png
 
abishai, vermaden: The configuration here actually boots from a UFS partition (where the kernel and other files in /boot are located), not a ZFS pool at all. For FreeBSD 10.1, the first stage UEFI boot loader only supports booting from a UFS partition.
 
vermaden said:
Because it uses TWO ZFS pools, one for boot pool and one for root pool.
Click to expand...
Not the reason. I looked into beadm quite ago, AFAIK it just replicates zroot/ROOT subsets. If something outside the scope, it is just left alone. So, according the guide only kernel is outside of the zroot scope (but, maybe one can solve the issue with symlink). beadm should work.
asteriskRoss said:
abishai, vermaden: The configuration here actually boots from a UFS partition (where the kernel and other files in /boot are located), not a ZFS pool at all. For FreeBSD 10.1, the first stage UEFI boot loader only supports booting from a UFS partition.
Click to expand...
Ah, I see. So, if I'm right, only a manually edit of vfs.root.mountfrom hint is needed? But that's not so bad, 90% of beadm will work.
 
abishai said:
Ah, I see. So, if I'm right, only a manually edit of vfs.root.mountfrom hint is needed? But that's not so bad, 90% of beadm will work.
Click to expand...

asteriskRoss did mention earlier in the thread that this guide is a little out-of-date, but for clarity's sake: vfs.root.mountfrom shouldn't be needed from 10.3 onward. (I believe the legacy bootloaders haven't needed it since 10.1, either.) You can instead run, per this guide's example set-up, zpool set bootfs=zroot/ROOT/master and the loader will automatically detect it. What's more, sysutils/beadm is just a convenience tool. You can manually replicate its behavior with relative ease:
  1. Create a snapshot of the current zroot/ROOT/master dataset, with a name representing the current system state (e.g. 10.3-RELEASE-p5).
  2. Clone that snapshot.
  3. Perform your upgrade, dangerous system tweak, whatever.
  4. If something goes wrong, change to the new boot environment by setting the bootfs property of the pool to the clone (e.g. # zpool set bootfs=zroot/ROOT/10.3-RELEASE-p5).
  5. Reboot.
Since this can be done manually, you can use boot environments with any dataset arrangement you want. But beadm is pretty sweet, and it's also sweet that the UEFI loader and menu support it now. I have to admit I use it myself. ;)
 
Thank you...
I did all of your instructions but when the system is rebooting, I've got this error :
No ZFS pools located, Can't boot.
Any idea?
 
Hi harika1258. I've been intending to write an updated version of this post for some time since as of FreeBSD 10.3 the UEFI bootloader supports booting from a ZFS pool, not just a UFS partition. Hooray!

Also a cause for celebration is the fact that the installer now includes the option to install to a ZFS pool in a GELI container so (as long as you're happy with the default encryption options; last I looked it used 256-bit AES-XTS which is fine if likely to be overly cautious for most users) it's possible to have an encrypted ZFS root without any of the manual and complex effort described in this HOWTO.

I see you have made a number of ZFS and GELI related posts since this one. If you're still looking for assistance perhaps you could send me a message to say which thread is still active and I'll do my best to help.
 
You must log in or register to reply here.
Back
Top