HOWTO: FreeBSD ZFS Madness

Guys,

Just an important update about ZFS+GRUB. After I've updated my FreeBSD 10-RELEASE to STABLE my grub-install doesn't work anymore. Always got the message unknown filesystem when I run it. I realized that happened after I updated my zpool with some feature flags, like hole_birth. It's incompatible with grub and will break your boot loader. So, do not update your zpool with feature flags until someone fixes it. It's important to say that on FreeBSD 10-STABLE feature flags are enabled by default, so if you want to create a new pool without feature flags use arg version=28, like this: zpool create -f -o version=28 -o altroot=/mnt -m none zroot /dev/ada0p3

Hope it helps someone else.
 
I've been testing beadm for some days, it's really a cool tool, but i'm worried with a real situation that i can face with. Let's suppose for some reason after i updated my system it doesn't boot anymore and i didn't activate my BE. How can i deal with this situation?

Is there a way that i can set on GRUB to boot it without beadm activate? I configured my grub boot menu to boot my BE, but it didn't work because it didn't find kernel to load.

To sum up, i just i'd like to choose on my grub boot menu which ZFS BE i'll boot without run beadmin activate each time.
 
laptop freebsd 10
Code:
zpool create -f -o cachefile=/tmp/zpool.cache sys /dev/gpt/sys0

Doesn't work.

Code:
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
cannot mount 'sys': No such file or directory

Code:
ls /dev/gpt
boot*** local *** sys0 <- ok

pleas help
 
unrealx0 said:
laptop freebsd 10
Code:
zpool create -f -o cachefile=/tmp/zpool.cache sys /dev/gpt/sys0

Doesn't work.

Code:
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
cannot mount 'sys': No such file or directory
The command DOES work.
Type zpool list after executing it.

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf
This is warning/notice about your system having less then recommended 4 GB for ZFS with Prefetch enabled.

ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
This is information about ZFS versions upon creation.

cannot mount 'sys': No such file or directory
You are doing install from Live CD, so / is read only and /sys can not be created, thus this warning.
 
thanks, now understands

next problem

Code:
zfs set mountpoint=/home local/home
Code:
cannot mount '/home': failed to create mount point property may be set but unable to remount filesystem
 
unrealx0 said:
thanks, now understands

next problem

Code:
zfs set mountpoint=/home local/home
Code:
cannot mount '/home': failed to create mount point property may be set but unable to remount filesystem
You are not doint steps in the right order, You should xecute that command after the zfs umount -a command.
 
Hi,

I just searched the forums up and down for information regarding using beadm with encrypted root ZFS pool. I find it mentioned that it's doable, although the standard procedure of FreeBSD's installer of creating an additional bootpool seems not to work. What I can't find after skimming this thread and the search function is a way to actually accomplish it. How do I set[]up a FreeBSD install with encrypted root ZFS pool that is still manageable via beadm?

Thanks for any help.
 
Thank you for this fast response. But I already saw that link, and it didn't work, so I assumed it was superseded by some other method.
When I download that script and use it instead of the version from pkg, I get the same "zpool.cache no such file" error as with the pkg version, only this time it doesn't exit and echos "Activated successfully". But after a reboot it still boots default instead of the activated one, which is still set to "R".
 
@yggdrasil

The boot pool idea is a dirty hack/workaround anyway, FreeBSD Devs should at last add support to loader to boot from GELI encrypted ZFS pool ... and implement Boot Environment support in the loader, till that time then only poosible solution is GNU GRUB2.
 
vermaden said:
@yggdrasil

The boot pool idea is a dirty hack/workaround anyway, FreeBSD Devs should at last add support to loader to boot from GELI encrypted ZFS pool ... and implement Boot Environment support in the loader, till that time then only poosible solution is GNU GRUB2.

Direct boot from a GELI encrypted ZFS pool will never be possible. There's just no way you can tell the boot loader how to decrypt the pool contents unless you have some additional filesystem (or other external method to obtain the keys) with the kernel and the decryption keys.
 
I, too, don't see direct boot from GELI encrypted pool coming anytime soon.
The best solution of course would be to fix beadm to work correctly with a bootpool, and make the freebsd loader work with BEs.

I very much dislike grub, and after just trying to install it in a VM and not getting it to work doesn't make that any better.

Is anyone here actually using GELI encrypted root on ZFS with boot environments?
 
Booting directly from an encrypted ZFS pool is never going to happen. It would be incredibly messy trying to build GELI code into the boot loader and quit likely impossible. I also see no reason to create all this mess just to allow a few generic system files to be encrypted. Why do people actually want this? Is it just for simplicity so you only need one pool?

What makes a lot more sense, and will probably happen at some point (although it appears no time soon), is to add Oracle ZFS style encryption to OpenZFS so that we can install FreeBSD to a pool, then create additional encrypted datasets on the same pool at will, to store whatever sensitive data we want encrypted.
 
Anyway, you can read my experience with GRUB2 and some tips on it.
Interesting, you were able to create the bios-boot partition using -t bios-boot. I had to use the command #gpart add -t \!21686148-6449-6E6F-744E-656564454649 -s <size> -i <index> <geom>.

Another thing I had to do was to create a "custom" GRUB image including both zfs and part_gpt modules (not there by default): grub-install --modules="part_gpt zfs" /dev/ada0, without which all I could get was the grub rescue prompt, which is almost useless.

...Whether or not you need the above, you will definitely have to create your own /boot/grub/grub.cfg file by hand, as grub-mkconfig can't create a working one. The truth is, this UUID stuff grub-mkconfig is trying to figure out is not going to be of any use, because grub2 just can't figure out ZFS pool using these UUIDs correctly. But if you point it to the right partition, you can see your ZFS pool file structure.

After following this advice I've managed to get sysutils/grub2-pcbsd to work in so far that I have a boot menu but I can only boot into the already activated BE. Attempting to select any other BE causes the system to hang partway through the boot process.

I think the cause of the problem is that beadm sets canmount=on for the children of the active BE and canmount=noauto for the others. If I attempt to boot into a non-activated BE then I end up mounting the root of the selected BE and the children of the previously activated BE.

The ability to select boot environments with GRUB would be ideal but it looks like further work is needed to fully integrate GRUB and beadm.

I think full integration might be possible it the following things are done:
  • Modify beadm to set canmount=noauto for all child datasets if ${GRUB} is set.
  • Provide an rc script to mount all child datasets for the selected boot environment.
  • Fix grub-mkconfig to work properly with ZFS. When beadm creates or deletes a BE it needs to run grub-mkconfig to keep the boot menus up to date so relying on a hand crafted /boot/grub/grub.cfg will not be a viable option
I'm tempted to start experimenting with a few scripts to try this out but I'd be interested in hearing any opinions before I start.
 
After following this advice I've managed to get sysutils/grub2-pcbsd to work in so far that I have a boot menu but I can only boot into the already activated BE. Attempting to select any other BE causes the system to hang part way through the boot process.

I think the cause of the problem is that beadm sets canmount=on for the children of the active BE and canmount=noauto for the others. If I attempt to boot into a non activated BE then I end up mounting the root of the selected BE and the children of the previously activated BE.

The ability to select boot environments with GRUB would be ideal but it looks like further work is needed to fully integrate GRUB and beadm.

I think full integration might be possible it the following things are done:
  • Modify beadm to set canmount=noauto for all child datasets if ${GRUB} is set.
  • Provide an rc script to mount all child datasets for the selected boot environment.
  • Fix grub-mkconfig to work properly with ZFS. When beadm creates or deletes a BE it needs to run grub-mkconfig to keep the boot menus up to date so relying on a hand crafted /boot/grub/grub.cfg will not be a viable option
I'm tempted to start experimenting with a few scripts to try this out but I'd be interested in hearing any opinions before I start.
My opinion hasn't changed. Investing time in a third-party port (GRUB) under the GPL2/GPL3 license instead of creating a native FreeBSD solution in the FreeBSD Loader is a waste of time.
 
I just updated the beadm - https://github.com/vermaden/beadm/tree/1.2.8 - to 1.2.8 version.

I happened to have zfs filesystems without any zfs properties included in my BE. Your new version embeds all property values it finds into double quotes like:
Code:
-o snapdir="hidden"
Now, without any zfs property available beadm will add the following to ...
Code:
zfs clone -o =""
... which will fail to create a new BE, e.g.:
Code:
cannot create 'zp0/ROOT/B1/usr/local': invalid property ''
The previous version just ran ...
Code:
zfs clone -o =
... which did not fail.

I did post a dirty patch in the FreeBSD mailing list:
Code:
--- beadm-1.2.8    2018-07-07 16:17:19.231902000 +0200
+++ beadm-1.2.8-patched    2018-07-07 22:00:19.740611000 +0200
@@ -213,7 +213,7 @@
        local OPTS=""
        while read NAME PROPERTY VALUE
        do
-          local OPTS="-o ${PROPERTY}=\"${VALUE}\" ${OPTS}"
+          local OPTS="-o ${PROPERTY}=${VALUE} ${OPTS}"
        done << EOF
$( zfs get -o name,property,value -s local,received -H all ${FS} | awk '!/[\t ]canmount[\t ]/' )
EOF
Well, that is not elegant at all, but it did work for me before adding zfs properties to my zfs filesystems in question.

I just wanted to let you know about that issue, though.
 
Try that one and let me know how it behaves:
https://github.com/vermaden/beadm/blob/master/beadm

Good news, it works for zfs filesystems without properties:
Code:
root> beadm list -a
BE/Dataset/Snapshot                                    Active Mountpoint       Space Created

11r336083
  zp0/ROOT/11r336083                                   NR     /                 1.1G 2018-07-08 12:29
  zp0/ROOT/11r336083/_jails                            -      /usr/home/jails  72.5M 2018-07-08 12:29
  zp0/ROOT/11r336083/usr                               -      /usr             15.5G 2018-07-08 12:29
  zp0/ROOT/11r336083/usr/local                         -      /usr/local      840.0M 2018-07-08 12:29
  zp0/ROOT/11r336083/usr/src                           -      /usr/src          1.7G 2018-07-08 12:29
  zp0/ROOT/11r336083/var                               -      /var            634.1M 2018-07-08 12:29

root> zfs create zp0/ROOT/11r336083/zzz

root> beadm list -a
BE/Dataset/Snapshot                                    Active Mountpoint       Space Created

11r336083
  [...]
  zp0/ROOT/11r336083/zzz                               -      /zzz             88.0K 2018-07-08 16:11

root> zfs get -o name,property,value -s local,received -H all | grep zp0/ROOT/11r336083/zzz

root> beadm create B1
cannot create 'zp0/ROOT/B1/zzz': invalid property ''

root> ./beadm-1.2.8-fixed create B2
Created successfully
Thank you very much for that quick fix!
 
Ok, I will now make new 1.2.9 version then.

Thanks for pointing this out.

Please send an update to the FreeBSD Mailing Lists that it will be fixed in 1.2.9.
 
Back
Top