HOWTO: GELI+ZFS for whole system inc. root with boot from USB stick

[donnex]

HOWTO: GELI+ZFS for whole system inc. root with boot from USB stick

I got interested in this setup the other day and I'll try to explain how I did it so that others can try it out and give feedback. Most of this is probably already known and tested but I couldn't find any complete howto for the whole process. You setup and install everything from the FreeBSD install disk, no need for a second harddrive, temporary partitions or anything like that. Everything you need is one or more internal hard drives, an install CD/DVD and a USB stick for booting.

Before trying this method make sure your box and bios is able to boot from a USB stick. Also make sure that the USB stick you have is bootable. Mine worked at first try so I don't really know of any good way to test this.

In the end my box the system will have:
* All internal hard drives will be encrypted with GELI and ZFS will be used for everything, including / (root) running on top of GELI.
* All internal hard drives will be completely encrypted with GELI, no need for an unencrypted /boot/ since we use a USB stick for boot and GELI keys.
* Swap will be encrypted with GELI with one time key.
* On boot the system will load kernel and modules from the USB stick and you'll be asked for passphrases to the GELI disks before the system will run.
* It's only possible to boot from the USB stick, unencrypted of course, containing /boot/. The GELI encryption keys and a loader.conf for the boot loader will be placed at the stick. The USB stick is only needed at boot time, when the system has booted up you can safely remove this and store it in a secure location and nothing on the system will be bootable or accessible without this stick and your passphrases.

The method I describe can be used in different setups and configurations. This is just how I setup my box. Instead of a raid0 you could setup ZFS in mirror or raid5 or whatever you want your setup to be.

My setup consists of:

• ad0 320 GB - Will be split in two with a 4 GB ad0s1a for swap and the rest, ad0s1b for ZFS.
• ad4 73 GB - The whole disk will be added to the zpool.
• da0 2 GB USB stick - This is the usb stick I use for booting and key files.

Installation and setup
The steps:
1. Boot install disk and drop to Fixit shell
2. Setup the Fixit shell
3. Setup partitions/slices on the internal disks
4. Setup GELI on the disks.
5. Create the ZFS pool
6. Mount ZFS in the right place
7. Back in sysinstall change install location and do a custom Distribution install
8. Back to Fixit shell and setup the USB stick and some config files.
9. Reboot and done

First off download and boot the system with a FreeBSD install disk. I used the FreeBSD 7.1 DVD but I think that the CD disk1 will work too.

At the install menu the first thing you'll do is to drop to the Fixit shell. Most of the work will be done there. You'll find it at the main menu of sysinstall. Choose option number 2 with livefs from CD/DVD.

Before starting out make sure you've backed up all your disks. Everything on the disks will be wiped.

You can do almost everything from the Fixit shell but it needs some setting up in order to do what we want. A few symlinks is needed in order to load modules and to use some applications.

Setup network (this is not necessary but it's much easier to just scp in the config files than to edit them live from the Fixit shell).
I use a local dhcp server to get networking set up but you could probably do this just as fine manually with ifconfig etc.

mkdir /var/db/
dhclient em0

Create /var/db/ wich is needed for dhclient to run. Then run dhclient on your interface.

If you want to be able to use scp to copy in configs and backup keyfiles you also need to create a symlink for ssh.

mkdir /usr/bin/
ln -s /dist/usr/bin/ssh /usr/bin/ssh

Now we need to load the GELI and ZFS module. In order to do this we need some symlinks from the CD/DVD.

ln -s /dist/boot/kernel /boot/kernel
ln -s /dist/lib /lib
kldload geom_eli
kldload zfs

This step is optional but it's always a good idea to clean out all the old stuff from the disks. For security, easy of mind etc.

dd if=/dev/urandom of=/dev/ad0
dd if=/dev/urandom of=/dev/ad4
dd if=/dev/urandom of=/dev/da0

Here I write random data to all my internal disks and to the USB stick.

This step is optional too but it makes the fixit shell a lot easier to work with.
Tips: Put this in a file and scp it in to the shell and then use source file to load it.

tcsh
setenv LANG "en_US.UTF-8"
setenv HISTCONTROL "ignoredups"
setenv PAGER "less"
setenv PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/mnt2/usr/local/sbin:/mnt2/usr/local/bin:/mnt2/usr/sbin:/mnt2/usr/bin:/mnt2/sbin:/mnt2/bin"

set autolist
set nobeep
set prompt="[%n@%m:%/]%# "

alias ls "ls -G -A -F"

Here you're switching shell to tcsh and setting it up. As I said this is completely optional but makes the shell a little more friendly.

I split my first disk (ad0) with 4 GB for swap space and the rest for ZFS.

fdisk -I /dev/ad0
bsdlabel -w /dev/ad0s1
bsdlabel -e /dev/ad0s1

This tells fdisk to use the whole disk, don't worry about the "Geom not found" error. Then write and edit the bsdlabels. I changed size for a: to 4G, offset to 0 and fstype to swap as this will be my 4 GB swap space. I then added a new line with b: * * unused wich tells bsdlabels that b should take all the rest of the disk.

a: 4G   0   swap
b: *   *   unused   0   0
c: (don't touch or edit)

This gives me 4 GB of swap space (a) and the rest of the disk for GELI and ZFS (b).

We now need a temporary place to store our key files for GELI and to generate the key files.

mkdir -p /root/keys
dd if=/dev/random of=/root/keys/ad0s1b.key bs=128k count=1
dd if=/dev/random of=/root/keys/ad4.key bs=128k count=1

This is the key files wich we'll use for GELI.

Initialize GELI for the disks. -b for passphrase on boot, -K keyfile, -s blocksize and -l keylen

geli init -b -K /root/keys/ad0s1b.key -s 4096 -l 256 /dev/ad0s1b
geli init -b -K /root/keys/ad4.key -s 4096 -l 256 /dev/ad4

Enter your secret passphrase twice for each geli init, this is needed at boot time in order to use the disks.

Now we attach them in order to use them.

geli attach -k /root/keys/ad0s1b.key /dev/ad0s1b
geli attach -k /root/keys/ad4.key /dev/ad4

If you check /dev/ you'll se that two new devices just showed up. /dev/ad0s1b.eli and /dev/ad4.eli in my case. This is the devices we'll use for ZFS.

We now need to create /boot/zfs/ because the zpool.cache is need on the USB stick when we boot. I learned this the hard way. Without this file ZFS won't work at boot and the system can't mount root.

mkdir /boot/zfs/

We only create this dir for now. ZFS will place a cache file there wich we later copy to the USB stick.

Now we can setup ZFS. As I said earlier my setup was just two disks in a raid0. Read more about ZFS on the FreeBSD wiki.

zpool create tank ad0s1b.eli ad4.eli

You can verify that the zpool got created with zpool status. It should show that it's using our .eli GELI disks. Don't forget the .eli extensions when setting up the zpool.

Now change mountpoint to legacy for our zfs tank. We'll mount ZFS manually.

zfs set mountpoint=legacy tank

Create the installation dir, mount my tank/zpool, create var tmp and usr (for mount options later like noexec etc) and mount them too.

mkdir /mnt/install/
mount -t zfs tank /mnt/install
cd /mnt/install/
mkdir var tmp usr
zfs create tank/var
zfs create tank/tmp
zfs create tank/usr
mount -t zfs tank/var /mnt/install/var
mount -t zfs tank/tmp /mnt/install/tmp
mount -t zfs tank/usr /mnt/install/usr

Continued in next post.....

[donnex]

....continue from previous post

Now it's time to install the base system. I always do a minimal install and add stuff later that I need. Exit the Fixit shell and return to the main sysinstall menu.

Go to Options and change Install Root to /mnt/install, where we mounted our newly created GELI+ZFS setup.
From the main menu go to Configure -> Distributions. I use:

• base
• kernels -> GENERIC
• man
• src -> base

When done select OK and install from CD/DVD and wait for it to complete the installation.

Now go back to the Fixit shell (CD/DVD), ignore the warnings. We'll now finish of the installation and setup our USB stick for booting the system.

Start off by setting up the USB stick for booting, install boot blocks etc. My USB stick is da0.

fdisk -BI /dev/da0
bsdlabel -B -w /dev/da0s1
newfs /dev/da0s1a
mkdir /mnt/usbboot/
mount /dev/da0s1a /mnt/usbboot

The first thing you need to do is to copy the boot dir from the install to the USB stick. Then rename GENERIC to kernel so that the boot loader will find it.

cp -Rpv /mnt/install/boot /mnt/usbboot/
cd /mnt/usbboot/boot/
rmdir kernel
mv GENERIC kernel
cd /mnt/install/boot/
rm -rf GENERIC

The last two steps is not really needed but the kernel on our ZFS /boot/ won't never be used and you'll probably build a new kernel when the system is up and running. Don't forget to remove the empty kernel dir, on the USB stick boot/. If you do your kernel will end up in kernel/GENERIC/ instead of just kernel.

Now we need to put our GELI keys on the USB stick. I put them in boot/keys/. We also need the zpool.cache file on the USB stick.

cd /mnt/usbboot/boot/
cp -r /root/keys /mnt/usbboot/boot/
cp /boot/zfs/zpool.cache /mnt/usbboot/boot/zfs/

The final step with the USB stick is to setup the loader.conf where you tell the boot loader to load geom_eli for GELI, load ZFS, where to find the GELI key files and where to find root. It must be placed in the boot dir on the USB stick (/mnt/usbboot/boot/loader.conf).

geom_eli_load="YES"
geli_ad0s1b_keyfile0_load="YES"
geli_ad0s1b_keyfile0_type="ad0s1b:geli_keyfile0"
geli_ad0s1b_keyfile0_name="/boot/keys/ad0s1b.key"
geli_ad4_keyfile0_load="YES"
geli_ad4_keyfile0_type="ad4:geli_keyfile0"
geli_ad4_keyfile0_name="/boot/keys/ad4.key"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank"

Change it for your system and disks. Also remember that tank is the name for my zpool, if you used another name you'll have to change this.

Now your USB stick is finished. You've copied /boot/ into it from the install, renamed GENERIC to kernel so that the boot loader will find it, copied in zpool.cache so that the zpool will be available at boot, set up the loader.conf and copied the GELI keys.

Now only two more files is needed. That's the local fstab file to mount the file systems at boot and rc.conf to add GELI to the swap.

When the system boot the loader.conf on the USB stick will tell the kernel that zfs:tank is our root. When that's done it's the local fstab file that make sure our swap and the other file system gets mounted. Go to /mnt/install/etc/ and edit fstab. This is what mine looks like.

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad0s1a.eli         none            swap    sw              0       0
tank                    /               zfs     rw              0       0
tank/var                /var            zfs     rw              0       0
tank/tmp                /tmp            zfs     rw              0       0
tank/usr                /usr            zfs     rw              0       0

This mounts all the zfs file systems we created earlier and also setup a GELI swap wich we need to add to rc.conf with a random one time key.

The last thing we need to do is to edit the rc.conf file to setup our swap. Add this into /mnt/install/etc/rc.conf.

geli_swap_flags="-e aes -l 256 -s 4096 -d"

You're now all done. Reboot the system and tell the BIOS to boot from the USB stick. If everything went well your system will boot up and ask you for the GELI passphrases and attach the encrypted drives. After that it'll mount root from ZFS, mount the other ZFS file systems and setup an encrypted swap.

When the system is fully booted you can remove the USB stick from the system and keep it in a secure location. It won't be possible to boot or access anything on the system without it so keep it safe. It's probably a good idea to backup the USB stick and the keys (Off site GPG encrypted image maybe?).

Now you'll probably want to do some cleanup and such. Set a root password, run tzsetup to set time zone etc. and the normal post install stuff.

If and when you rebuild your kernel you'll have to copy it to the USB stick and overwrite what's there already. Remember to make sure that loader.conf and the keys folder is still there, otherwise the system won't boot.

Other info

• You can verify that your swap is working and is GELI encrypted by checking swapinfo. It should say disk.eli.
• Remember that you can scp files into the fixit shell if you setup networking and the necessary symlinks. It's much easier to edit files and configs on a remote box and then scp them into the install.
• ZFS on FreeBSD is NOT considered stable yet. You'll probably need some tweaking to get it working correctly. See http://wiki.freebsd.org/ZFS
• If the install wasn't successful or you missed a step you can boot the install CD/DVD and enter the Fixit shell. Create the necessary symlinks and load the modules. Now you can attach your GELI devices and access the ZFS zpool. If no pool is visible when you run zpool list, try zpool import and zpool import tank.
• If you change your ZFS setup, add disks etc, you probably need to copy the new /boot/zfs/zpool.cache from the local file system to the USB stick. You'll also need to setup GELI on the new disk and add key file + change loader.conf on the USB stick.
• The same method can be used to install and setup a system with GELI but without ZFS. All you have to do is to edit the bsdlabels correctly, skip the ZFS stuff, use newfs to create filesystems and then edit the fstab and configs.
• I know that raid0 is dangerous. I used it for testing purposes only on the box.

This is what I come up with after some reading and a few failed attempts. Any feedback, suggestions or comments is more than welcome.

[Carpetsmoker]

A few comments:

The integrity of the data on the USB disk is not checked, for all you know someone compromised the kernel on our USB stick, making the disk encryption useless.
I would recommend storing a one-way hash such as SHA2 of every file on a encrypted partition, and check the integrity of with a rc.d script as soon as possible in the boot process.
I've been wanting to do this on my system for some time, I'll see if I have time this weekend...

You don't need to use an USB stick, you can use a separate `boot slice' instead, which works more or less in the same manner, just like the bootable USB stick it just needs to contain /boot/ and /etc/fstab.
I can't think of any advantages of using a USB stick, other than that you can only multi-boot 3 OS's instead of 4 now.

I did some symlinking to make stuff like kldload work, my boot disk contains /boot/* and /etc/fstab.
I mount this at /boot2, and then symlink /boot2/boot to /boot and /boot2/etc/fstab to /etc/fstab.
This is not really needed, but I find it useful.

[killasmurf86]

Originally Posted by Carpetsmoker

A few comments:
You don't need to use an USB stick, you can use a separate `boot slice' instead, which works more or less in the same manner, just like the bootable USB stick it just needs to contain /boot/ and /etc/fstab.
I can't think of any advantages of using a USB stick, other than that you can only multi-boot 3 OS's instead of 4 now.

Imagine you come home and your PC with FreeBSD with all the torrents are missing Some FBI (or what ever) took it as evidence.....
If you keep keys (that is if you use them at all) on separate boot partition, all they have to do is just crack your password (which is much easier. [that is i assume you use password as well])
But if you carry your flash with you... they won't have keys...

btw, you can keep keys on flash and still boot from separate partition on HDD

[Carpetsmoker]

Imagine you come home and your PC with FreeBSD with all the torrents are missing Some FBI (or what ever) took it as evidence.....

Logging out or locking the system will prevent this ... This is something you should always do in any case.

If you keep keys (that is if you use them at all) on separate boot partition, all they have to do is just crack your password (which is much easier. [that is i assume you use password as well])

``All they have to do is just crack your password'' :o

I would like to see the FBI (or NSA) crack my (strong) 60 character passphrase. The best known attack for AES is brute-force.
Cracking a strong passphrase is nowhere near easy.

[lme@]

Originally Posted by Carpetsmoker

I would like to see the FBI (or NSA) crack my (strong) 60 character passphrase. The best known attack for AES is brute-force.

At least they know now that your passphrase contains 60 chars.

[Carpetsmoker]

Originally Posted by lme@

At least they know now that your passphrase contains 60 chars.

It doesn't contain exactly 60 characters ... Not even about actually. In fact, I pulled the number out of my arse.

[DutchDaemon]

Damn, it's hard to respond to that without using the word 'crack'.

So I won't.

[Carpetsmoker]

Originally Posted by DutchDaemon

Damn, it's hard to respond to that without using the word 'crack'.

So I won't.

Ehm ... I think this says more about you than it says about me ... :-/

[mfaridi]

I wish I can find very easy how to for use ZFS for whole of system.

[lme@]

Here: http://rink.nu/entry.php?id=105 (ZFS + GELI with UFS /boot)
or here: http://lulf.geeknest.org/blog/freebs...s-only_system/
(ZFS only with GPT)

[killasmurf86]

I tried ZFS on root (with geli)....
It was crashing every few minutes....
My record was 15min without crash

however i'm running i386 with 1.5G ram, maybe that's the case
[FreeBSD-7.2]

[Djn]

Originally Posted by Carpetsmoker

Logging out or locking the system will prevent this ... This is something you should always do in any case.



``All they have to do is just crack your password'' :o

I would like to see the FBI (or NSA) crack my (strong) 60 character passphrase. The best known attack for AES is brute-force.
Cracking a strong passphrase is nowhere near easy.

Not to be unpleasant - but this seems relevant.

[lme@]

Originally Posted by killasmurf86

I tried ZFS on root (with geli)....
It was crashing every few minutes....
My record was 15min without crash

however i'm running i386 with 1.5G ram, maybe that's the case
[FreeBSD-7.2]

I was running ZFS + Geli on my i386 notebook with 8-CURRENT successfully for months. There was only 1GB in it, but I didn't have any ZFS-related crashes. Now I upgraded the box to 2.5 GB now just to have some more RAM for non-ZFS related stuff.
I'd suggest upgrading your machine to 8-CURRENT and tune your system as described in the ZFS tuning guide in FreeBSD's wiki.

[killasmurf86]

Originally Posted by lme@

I was running ZFS + Geli on my i386 notebook with 8-CURRENT successfully for months. There was only 1GB in it, but I didn't have any ZFS-related crashes. Now I upgraded the box to 2.5 GB now just to have some more RAM for non-ZFS related stuff.
I'd suggest upgrading your machine to 8-CURRENT and tune your system as described in the ZFS tuning guide in FreeBSD's wiki.

http://forums.freebsd.org/showpost.p...5&postcount=23