ZFS: Boot from USB

markfisher · Apr 9, 2014

Hi

I'm experimenting with ZFS file system to learn more about it. I'm trying to boot my system from USB (having /boot partition in USB and rest in HDD). For simplicity no encryption involved.

USB Stick: da0
HDD: ada0

Here is the steps I took (Boot from DVD (FreeBSD 10) and select shell, also /mnt is Read/Write):

Code:

gpart create -s gpt da0
gpart create -s gpt ada0

gpart add -a 4k -s 512K -t freebsd-boot da0
gpart add -a 4k -t freebsd-zfs da0

gpart add -a 4k -s 2G -t freebsd-swap ada0
gpart add -a 4k -t freebsd-zfs ada0

gpart set -a bootme -i 2 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Creating ZFS Pools:

Code:

zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache usbboot /dev/da0p2
zpool create -o altroot=/mnt -o cachefile=/tmp/zpool.cache hddroot /dev/ada0p2

zfs set mountpoint=none usbboot
zfs set mountpoint=none hddroot

zfs set checksum=fletcher4 usbboot
zfs set checksum=fletcher4 hddroot

Creating Data Set:

Code:

zfs create -o mountpoint=none hddroot/ROOT
zfs create -o mountpoint=/ hddroot/ROOT/default
zfs create -o mountpoint=/home hddroot/home
zfs create -o mountpoint=/usr hddroot/usr
zfs create -o mountpoint=/var hddroot/var
zfs create -o mountpoint=/tmp hddroot/tmp

chmod 1777 /mnt/tmp

zfs create -o mountpoint=/boot -o compression=off usbboot/boot

zpool set bootfs=hddroot/ROOT/default hddroot

Install FreeBSD:

Code:

cd /usr/freebsd-dist
for i in base kernel src ports; do
xz -d -c $i.txz | tar -C /mnt -xf -
done

chroot /mnt
touch /etc/rc.conf
touch /etc/fstab
touch /boot/loader.conf

exit
cp /tmp/zpool.cache /mnt/boot/zfs/
shutdown -r now

My /etc/rc.conf, /boot/loader.conf, and /etc/fstab

Code:

# /etc/rc.conf
zfs_enable="YES"

# /boot/loader.conf
zfs_load="YES"

# /etc/fstab
/dev/ada0p1    none    swap    sw    0    0

My problem is when I reboot, my system won't boot, it's looking for /boot/kernel/kernel. Does anyone knows what I did wrong on my ZFS setup?

Thanks

asteriskRoss · Apr 15, 2014

I use a similar set up on my laptop so that I can use a GELI encrypted ZFS root. You're close!

A combination of the man pages for gpart(8) and zpool(8) tell us what the boot loader is looking for. From the man page for gpart(8):

/boot/gptzfsboot is used to boot from ZFS. It searches through the GPT for freebsd-zfs partitions, trying to detect ZFS pools. After all pools are detected, /boot/zfsloader is started from the first one found.

Note that unlike the description for /boot/gptboot there is no reference to looking at flags, so I wouldn't expect the "bootme" flag you set to have any effect.

From the man page for zpool(8):

bootfs=pool/dataset
Identifies the default bootable dataset for the root pool. This property is expected to be set mainly by the installation and upgrade programs.

You want to boot from a dataset on the USB memory stick, not the hard disk, so you need to set the "bootfs" property on your usbboot pool to the appropriate dataset. If you were booting from the usbboot/boot dataset:

Code:

[del]zfs[/del]zpool set bootfs=usbboot/boot usbboot

You also need to restructure this dataset. Remember that there is no root filesystem mounted yet, so setting a /boot mount point as a ZFS property won't yet be useful. This can be a little confusing. The way I think about it is that the boot loader (gptzfsboot) is expecting to be booting from a typical FreeBSD installation, where the boot directory is on the root partition. So it will be looking at the dataset you specified in the "bootfs" property for a directory called "boot". At the moment, you don't have one, since you installed the contents of the /boot directory straight into your usbboot/boot ZFS dataset. To fix that, import your usbboot pool with an alternative root of /mnt, which if you have default settings will likely automount your usbboot/boot ZFS dataset at /mnt/boot:

Code:

zpool import -f -R /mnt usbboot

Create the boot directory:

Code:

cd /mnt/boot
mkdir boot

Use tar(1) to move the contents into the right place:

Code:

tar cf - --exclude boot . | ( cd boot && tar xvd - )
rm `ls | grep -v boot`

You may have noticed that this messes up your plan to have the boot files mounted in the correct place once your system is up and running, as they will now end up in /boot/boot. I suggest changing the mount point to something like /usbboot and then linking /boot to that directory:

Code:

zfs set mountpoint=/usbboot usbboot/boot

Once your system is up and running (we're not there yet):

Code:

ln -s /usbboot/boot /boot

Lastly, you need to tell the boot loader where to find the root partition, since it isn't the one that contains the boot files and kernel. Edit your loader.conf (now in the boot directory of the usbboot/boot dataset) and add:

Code:

vfs.root.mountfrom="zfs:hddroot/ROOT/default"

To clean up you should probably also unset the "bootfs" property on your hddroot pool since you won't be booting from it (you may need to alter the syntax on this one as I haven't tested it):

Code:

zpool set bootfs=none hddroot

markfisher · Apr 16, 2014

@asteriskRoss
Thanks for your reply. I followed your instruction and everything is working now without encryption.

I have one question:
Following your instruction, I'm now trying to encrypt my hddroot without requiring to input passphrase. I looked at geli(8), and:

Code:

geli init -K /tmp/hddroot.key -P /dev/ada0p1
geli attach -k /tmp/hddroot.key -p /dev/ada0p1

I copied the /tmp/hddroot.key to /boot and add (/boot/loader.conf):

Code:

zfs_load="YES"
crypto_load="YES"
geom_eli_load="YES"

geli_ada0p1_keyfile0_load="YES"
geli_ada0p1_keyfile0_type="ada0p1:geli_keyfile0"
geli_ada0p1_keyfile0_name="/boot/hddroot.key"

Root won't get mounted. I also tried (without -P and -p):

Code:

geli init -K /tmp/hddroot.key /dev/ada0p1
geli attach -k /tmp/hddroot.key /dev/ada0p1

Root still won't get mounted. Do you know how to encrypt the drive without requiring passphrase?
Please let me know if you need more information regarding my setup.

Thank you.

asteriskRoss · Apr 16, 2014

Try setting the boot flag on the GELI provider so it is attached before the root filesystem is mounted. From the geli(8) man page:

init -b
Ask for the passphrase on boot, before the root partition is mounted. This makes it possible to use an encrypted root partition. One will still need bootable unencrypted storage with a /boot/ directory, which can be a CD-ROM disc or USB pen-drive, that can be removed after boot.

To reconfigure your existing provider (from memory I think you need to manually attach it first):

Code:

geli attach -k /path/to/geli/keyfile/hddroot.key /dev/ada0p1
geli configure -b /dev/ada0p1

Alternatively, include the -b flag when you initialise the GELI partition:

Code:

geli init -b -P -K /path/to/geli/keyfile/hddroot.key /dev/ada0p1

markfisher · Apr 18, 2014

@asteriskRoss

Thanks for your help. Your solution works perfectly.

Thank you.

Martijn007 · Apr 3, 2015

Hi Ross,

i would like to ask you if you could get me up to speed with similar things.
I happen to have an Asus board (UEFI) which does not work very well on BSD's GPT scheme (change to windows, according to support).

I would like to boot from USB with MBR.

Is daemon-notes your site?

I would like to know how to setup the stick with only the boot-partition (minimal writes) to mount the GPT zfs pool with the rest of the OS.

Thanks!

asteriskRoss · Apr 3, 2015

Hi Martijn007

I haven't seen this thread for a while!

Martijn007 said:
Is daemon-notes your site?

No, that's not mine.

Martijn007 said:
i would like to ask you if you could get me up to speed with similar things.

Absolutely and I'm sure other forum users will pitch in too.

I was going to ask whether you already tried setting the active flag on the protective MBR as per Thread freebsd-gpt-uefi.42781, which shouldn't be needed but seemingly some UEFI implementations expect it, but I see you already posted in Thread landing-in-bios-again-and-again.46016 saying that hadn't worked.

For some background/reference, you might find the following useful:

FreeBSD Booting Process section of the handbook
gpart(8) man page
boot0cfg(8) man page.

If I understand correctly, you want to set up your computer to boot using legacy BIOS, rather than UEFI and configure your USB stick with necessary partitions and files to mount and use a ZFS dataset containing your root filesystem. You didn't mention GELI, so I'm going to assume you're not using it, though if you are the only part to change would be the loader.conf(5) configuration.

For the first part, you'll need to dive into your computer's firmware settings. Some firmware forces you to choose legacy BIOS or UEFI boot, other firmware allows you the option of both at the same time (my laptop lets me do this and press a key to choose my boot options) and other firmware doesn't support legacy BIOS at all.

For the second part, you'll need to partition your USB stick and put the boot loader files in the right place. Boot into the FreeBSD installation CD or memory stick and choose the Live CD option. Plug in the USB stick you want to configure and take a note of what device it is assigned; likely/dev/da0 or /dev/da1. I'll assume the latter and also that you have already copied off any data from that USB stick you want to keep. Let me also warn you that I've not tried this (I use GPT scheme) so please don't assume any mistakes are yours but post how you get on and we can figure it out.

Destroy whatever partitioning scheme is configured for your USB stick:
gpart destroy -F /dev/da1

Create a MBR partition scheme:
gpart create -s MBR /dev/da1

Create a FreeBSD slice and mark it active:

 gpart add -t freebsd /dev/da1

gpart set -a active -i 1 /dev/da1

Install the boot0 boot manager:
gpart bootcode -b /boot/boot0 /dev/da1

Create a BSD label inside the slice:
gpart create -s BSD /dev/da1s1

Create a UFS partition, which is where your /boot directory will live and also creat the UFS filesystem:

 gpart add -t freebsd-ufs /dev/da1s1

newfs /dev/da1s1a

Install the bootstrap code:
gpart bootcode -b /boot/boot /dev/da1s1

Now you need to mount the UFS partition and copy the /boot directory and contents. This copies from the installation media, but you could take from elsewhere if desired:

 mount /dev/da1s1a /mnt

mkdir /mnt/boot

(cd /boot && tar cf - .) | ( cd /mnt/boot && tar xvd - )

Your USB stick is now good to go, but you will need to configure loader.conf(5) to mount your root filesystem from the ZFS dataset on your hard disk. So create /mnt/boot/loader.conf to contain the following, substituting pool for the name of the ZFS pool on your hard disk and dataset for the ZFS dataset in that pool containing your root filesystem:

Code:

zfs_load="YES"
vfs.root.mountfrom="zfs:pool/dataset"

Reboot with shutdown -r now, press whatever you need to boot to the USB stick and cross your fingers!

Once you have it working, you might want to do some cleaning up. The /boot directory on your root filesystem is no longer used and might confuse later if you wonder why your boot configuration changes don't take effect. You could remove it, or ensure you copy any changes to the USB stick's boot directory.

You can create an entry in fstab(5) to allow you to mount /dev/da1s1a to /usbboot or similar. If you went with the remove option, then you could create a symbolic link for /boot that points to /usbboot/boot. If you mount it automatically, you will need to leave the USB stick plugged in but loading kernel modules, updating your kernel and boot configuration will work as it would on a typical system. If you choose to mount it manually, you will need to remember to do so when you need it.

Maintaining a copy of /boot on the hard disk isn't necessarily a bad option, but you will need to remember to copy all changes to the USB stick.

Martijn007 · Apr 8, 2015

Code:

vfs.root.mountfrom="zfs:starten/zfsguru/10.1-001"

led to

Code:

init initial setsid() failed operation not permitted

Then it prompted for the correct location of sh.

Any ideas?

asteriskRoss · Apr 9, 2015

If you boot to the installation media and choose Live CD, are you able to import your ZFS pool?
zpool import -f -R /mnt starten

Are the mountpoints of your ZFS datasets correctly configured?
zfs list -r -o name,canmount,mountpoint starten

Do you have the following line in rc.conf(5) (/mnt/etc/rc.conf from the LiveCD after successful ZFS pool import) to automatically mount ZFS datasets at boot?

Code:

zfs_enable="YES"

Martijn007 · Apr 10, 2015

I'll have a look at it.

Martijn007 · Apr 12, 2015

Hmm... I'm not sure if I'm following.

The steps it took now:

Boot from Live-CD, setup the pool on the normal drive(s), entered the shell to prepare the stick, edit the /mnt/boot/loader.conf to setup:

Code:

zfs_load="YES"
vfs.root.mountfrom="zfs:starten"

Rebooted with error.

Restarted Live-CD, imported pool, mounted stick again, changed:

Code:

vfs.root.mountfrom="zfs:starten" to
vfs.root.mountfrom="zfs:starten/zfsguru/10.1-001"

So yes, the pool is importable but I think I copied the boot-folder from the Live-CD instead of the pool.

Martijn007 · Apr 12, 2015

A lot to learn I still have...

I see that I forgot to copy the /etc folder.

Also I took the original loader.conf file from the Live-CD (?!)

From a working machine I copied the loader.conf, now the thing is working.

Thanks a lot.

I don't have the USB stick mounted, so is any change in rc.conf or loader gone when rebooted?

asteriskRoss · Apr 12, 2015

Martijn007 said:
I see that i forgot to copy the /etc folder.
[...]
I don't have the USB stick mounted, so is any change in rc.conf or loader gone when rebooted?

/etc should be on your root filesystem (on your ZFS dataset), not on the USB stick, so changes to /etc/rc.conf need not be copied elsewhere. To where did you need to copy it? There might be another issue here with your configuration -- did you put /etc in a separate ZFS dataset for example?

Martijn007 said:
Also I took the original loader.conf file from the Live-CD (?!)
From a working machine I copied the loader.conf, now the thing is working.

Great

The instructions I wrote suggested copying from the LiveCD, but I intended for you to edit the file afterwards

Remember that your kernel and kernel modules live in /boot, which is now on the USB stick, so if you update your kernel (either by recompiling or using freebsd-update(8)) you will need to make sure the changes happen in the right place. This is where having only a single copy of /boot, keeping the USB stick mounted (to /usbboot for example) and using a symbolic link to make it appear in the normal place on the filesystem ( ln -s /usbboot/boot /boot) makes management a lot easier.