Geli+ZFS on root (mirrored drives)

Y

yudi

Hi all,

I have gone through the wiki and countless other posts on the Internet, most of them list commands but offer very little information on why that command is being used or why a particular method is being used.

I am trying to write up the basic steps needed to set up as system with GELI+ZFS on root on mirrored drives. I would greatly appreciate your help in filling the gaps and correcting any errors.

I am mainly following: https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE

I an currently testing it in a VM and the set up is as follows:
  • 2 GB RAM
  • two HDDs 8 GB each mirrored - three partitions
    1. for boot code 128 KB
    2. for /boot 2 GB
    3. for the rest of the system and encrypted
  • no key file for encrypted partitions, only passphrase
  • using 9.1-RELEASE
  • there will be no swap or handling of 4k drives, just to keep it as simple as possible.

Create the basic three partitions:
Code: 
gpart destroy -F ada0
gpart destroy -F ada1
gpart create -s gpt da0
gpart create -s gpt da1
gpart add -s 128 -t freebsd-boot da0 
gpart add -s 128 -t freebsd-boot da1
gpart add -s 2G -t freebsd-zfs da0
gpart add -s 2G -t freebsd-zfs da1
gpart add -t freebsd-zfs da0
gpart add -t freebsd-zfs da1

Write boot code to both disks:
Code: 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

Load necessary modules:
Code: 
kldload opensolaris (I'm not sure why this is needed as loading zfs also loads opensolaris?)
kldload zfs
kldload geom_eli
Encrypt the disks with only a passphrase:
(I'm not sure why a lot of how-tos suggest using key files but then place them in /boot which is easily accessible, am I missing something?)
Code: 
geli init -b -s 4096 /dev/da0p3 
geli init -b -s 4096 /dev/da1p3
 
geli attach /dev/da0p3
geli attach /dev/da1p3

Creating ZFS datasets:
Code: 
zpool create bootdir mirror /dev/da0p2 /dev/da1p2
zpool set bootfs=bootdir bootdir
 
zpool create -R /mnt -O canmount=off tank mirror /dev/da0p3.eli /dev/da1p3.eli 
zfs create -o mountpoint=/ tank/ROOT
zfs set mountpoint=/mnt/ROOT/bootdir bootdir 
zfs mount bootdir

Then exit out of the shell and go back to bsdinstall. Install as normal and then get back to the shell after bsdinstall finishes. This is where I am not sure, I thought everything would be installed under tank/ROOT (as it was set as the /) but it got installed under /tank.

At the end of bsdinstall, it gives an option to get into the installed system to do final setting-up. Get back to the shell (this is the newly installed system and the PWD is tank dataset not ROOT, everything was extracted to tank not ROOT), then: mount -t devfs devfs /dev .
Otherwise one cannot use ZFS commands in the new environment.

Add the necessary variables/settings:
Code: 
echo ‘zfs_enable=”YES”‘ >> /etc/rc.conf
echo ‘vfs.root.mountfrom=”zfs:tank”‘ >> /boot/loader.conf (Is this correct? or is it needed?)
echo ‘zfs_load=”YES”‘ >> /boot/loader.conf
echo ‘geom_eli_load=”YES”‘ >> /boot/loader.conf

Then create a zpool cache file: zpool set cachefile=/boot/zfs/zpool.cache tank.

Then move the boot folder to the second partition under the bootdir dataset:
Code: 
mv boot ROOT/bootdir/ 
ln -fs ROOT/bootdir/boot

Then set the final mount points:
Code: 
zfs unmount -a (this throws an error, cannot unmount /mnt, it's busy)
zfs set mountpoint=legacy tank
zfs set mountpoint=/ tank/ROOT 
zfs set mountpoint=/bootdir bootdir

Exit and reboot.

That's the basic outline. I would greatly appreciate if you could point out errors and offer some explanation.

Cheers,
Yudi
 
yudi said:
I'm not sure why a lot of how-tos suggest using key files but then place them in /boot which is easily accessible, am I missing something?
Click to expand...
The keyfiles are typically placed on a removable USB flash drive, which is only used to boot the system and removed once the system has booted.

yudi said:
Then move the boot folder to the second partition under the bootdir dataset:
Click to expand...
You mean boot directory. Folders are for storing e-mail and for communicating with Windows people ;)
 
I tried the above steps and encountered the mountroot prompt. I can open the encrypted container and then I encounter the mountroot problem. I am sure the steps up to encrypting and attaching the geli partitions are correct. If there was an error it would have to be with the zpool config. I even checked the zpool.cache was present in /boot/zfs/ before I rebooted.

As I have mentioned above, I am using https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/9.0-RELEASE as a template. Trying to understand some of the commands in the above link:
zfs create -o mountpoint=/ zroot/ROOT
I am guessing ROOT is being created to use boot environments in future, right?

When setting the final mount points:
Code: 
# zfs set mountpoint=/ zroot/ROOT 
# zfs set mountpoint=/zroot zroot
What is the purpose of the second line above, why is the root dataset being mounted under ROOT?

Also the following value is not set in the scenario listed in the wiki link:
Code: 
echo ‘vfs.root.mountfrom=”zfs:pool”‘
Is the above value even necessary? Or is it necessary when there is more than one root dataset (in my case bootdir and tank). Or if it differs from the bootfs value? Theoretically, the only difference between unencrypted set-up and encrypted one is that the /boot is outside the encrypted container (or am I missing something). So all I am trying to do is follow the steps in the wiki links and use *.eli containers as vdevs for everything except /boot. Install everything to the pool on the *.eli containers and then move /boot to the unencrypted partition/pool. I cannot figure where I am going wrong.

Any help?
 
Success at last.

It was
Code: 
 vfs.root.mountfrom=”zfs:tank"
that was incorrect. It needs to be the filesystem that the system is installed on, not just the root dataset.

I changed it to
Code: 
vfs.root.mountfrom=”zfs:tank/ROOT”
and it loads without an issue.

From what I can tell in the wiki link above, the author used:
Code: 
 # zpool set bootfs=zroot/ROOT zroot
That's why he did not need the vfs.root.mountfrom option as all the system boot files were installed to the ROOT filesystem, but when using GELI bootfs is set to the unencrypted dataset as /boot needs to be unencrypted.

Code: 
 zpool set bootfs=bootdir bootdir
but [size=-1][Who, you or the wiki author? -- Mod.][/size] then had to use
Code: 
vfs.root.mountfrom=”zfs:tank/ROOT”
to point to the right filesystem where the rest of the system files are located. Am I right?

I would still like to know the answers to my previous questions.
 
yudi said:
Success at last.

but [size=-1][Who, you or the wiki author? -- Mod.][/size] then had to use
Code: 
vfs.root.mountfrom=”zfs:tank/ROOT”
to point to the right filesystem where the rest of the system files are located.
Click to expand...

The wiki author did not set vfs.root.mountfrom. I think it is optional when not using geli. Because ZFS bootfs flag does the job.

Most of the conclusions I reached are based on snippets of information found online and guess work/trial and error.
It would be nice for someone knowledgeable about this stuff to confirm and offer an explanation.

Cheers,
Yudi
 
I think the problem was using tank/ROOT in the first place. Usually, for installation you mount your tank somewhere (probably/e.g. called /mnt/ROOT in your installation file system), copy some files to it and the chroot into it to fix e.g. the vfs.root.mountfrom.
All these mountpoints (from the Wiki/tutorial I presume) also strike me as odd. I only have a single 'legacy' mountpoint that does the job. And with a legacy mountpoint you have to use vfs.root.mountfrom.
 
You must log in or register to reply here.
Back
Top