HOWTO: FreeBSD ZFS Madness

randomcop said:
Thanks yor your great Howto and the beadm script!

I have already done some testinstalls both under virtualbox and on some real server hardware and the zfs-on-root + beadm setup really works ok.

What really makes me nervous are thinkable situations, when I have an active boot environment that locks up right after the kernel is loaded or some other form of broken freebsd. How can I go back to a previous, stable boot environment, without being able to use the beadm script?


You will have to use FreeBSD LiveCD or LiveUSB and do it 'by hand'.

randomcop said:
As an experiment I created a new boot environment from default, named "be1". I activated "be1" with beadm and on startup escaped to loader prompt. There I did:

- unload kernel
- set vfs.root.mountfrom=zfs:sys/ROOT/default (from vfs.root.mountfrom=zfs:sys/ROOT/be1)
- set currdev=zfs:sys/ROOT/default: (from currdev=zfs:sys/ROOT/be1:)
- load kernel
- load zfs
- boot

This leads to the the result, that - after the kernel loaded ok - sys/ROOT/default cannot be mounted and the following text is displayed:

mounting from zfs:sys/ROOT/default: failed with error 2

This error comes mostly when zpool.cache is not up to date, to make it up to date it requires to export and import a pool.



randomcop said:
What can I do if I have a an active, but broken, boot environment, and want to revert to a previous, stable boot environment?


You will have to use FreeBSD LiveCD or LiveUSB and do it 'by hand' and with the zpool.cache 'regeneration' step.
 
kpa said:
There's a recent change on 9-STABLE that makes the zpool.cache optional for bootable ZFS pools. I think that beadm(1) should not consider a missing /boot/zfs/zpool.cache an error but print a warning that the pool may not be bootable unless the OS is recent enough.

http://forums.freebsd.org/showthread.php?t=36513

I have read about it, but not all parts are in the 9.1-STABLE yet:
The change was introduced via multiple commits, the latest relevant revision in
head is r243502. The changes are partially MFC-ed, the remaining parts are
scheduled to be MFC-ed soon.

Also, zpool.cache is used in 9.1-RELEASE, so beadm(1) will support it. I can check for FreeBSD version (r243107 for example) in beadm(1) and act accordingly with no zpool.cache for newer versions, but as I said, they are not even in STABLE yet ;)
 
@kpa

Nice to see these changes already in STABLE, I will have to test that out in some vm to check how the install procedure differs, what is not longer needed and what can be 'ifed' in beadm(1) to not touch zpool.cache again, thanks for clarification mate ;)
 
Hi,

I have read through this thread a couple of times, since I´ve wanted to implement BE on a new server I´m installing, but I think there will be something preventing it. The servers I´m administering are storage units that have two USB's configured with ZFS as mirrored / that are used to boot the machine. Then, other filesystems, like /usr,/usr/local,/usr/src,/usr/obj,/usr/ports etc, are configured up as part of the bigger pool, made up of the hard drives in the system. The layout I was going for would then look something like:
Code:
FS                                   MOUNTPOINT
pool0                                none
pool0/ROOT/9.1-RELEASE               legacy (bootfs="pool0/ROOT/9.1-RELEASE", vfs.root.mountfrom="zfs:pool0/ROOT/9.1-RELEASE")
pool0/ROOT/9.X-RELEASE               legacy
pool0/ROOT/X.Y-RELEASE               legacy
pool1                                none
pool1/ROOT/9.1-RELEASE               none
pool1/ROOT/9.1-RELEASE/usr           /usr
pool1/ROOT/9.1-RELEASE/usr/home      /usr/home
pool1/ROOT/9.1-RELEASE/usr/local     /usr/local
pool1/ROOT/9.1-RELEASE/usr/obj       /usr/obj
pool1/ROOT/9.1-RELEASE/usr/ports     /usr/ports
pool1/ROOT/9.1-RELEASE/var           /var
pool1/ROOT/9.X-RELEASE               none
pool1/ROOT/9.X-RELEASE/usr           /usr
pool1/ROOT/9.X-RELEASE/usr/home      /usr/home
pool1/ROOT/9.X-RELEASE/usr/local     /usr/local
pool1/ROOT/9.X-RELEASE/usr/obj       /usr/obj
pool1/ROOT/9.X-RELEASE/usr/ports     /usr/ports
pool1/ROOT/9.X-RELEASE/var           /var
pool1/ROOT/X.Y-RELEASE               none
pool1/ROOT/X.Y-RELEASE/usr           /usr
pool1/ROOT/X.Y-RELEASE/usr/home      /usr/home
pool1/ROOT/X.Y-RELEASE/usr/local     /usr/local
pool1/ROOT/X.Y-RELEASE/usr/obj       /usr/obj
pool1/ROOT/X.Y-RELEASE/usr/ports     /usr/ports
pool1/ROOT/X.Y-RELEASE/var           /var
tmpfs                                /tmp
pool1/SWAP                           -
pool1/EXPORT                         none
pool1/EXPORT/datastore               /export/datastore
Would beadm be able to cope with this type of layout?

Also, since these systems are booting from USB, I´ve configured the systems to mount / as read-only to prolong their lives by quite far. What whould be the best-practice way to configure that in use with beadm? Up until now, I´ve only used plain-ol' fstab with all fs's legacy to handle things, so I´m quite new to ZFS´s automagic:)

/Sebulon
 
Sebulon said:
Would beadm be able to cope with this type of layout?

The beadm operates on ${BOOTPOOL}/ROOT/${BENAME} and anything that is below that path is taken into the Boot Environment. Anything else, even with the same 'schema' is omitted ${OTHERPOOL}/ROOT/${BENAME} will not be snapshotted and used for beadm.

Sebulon said:
Also, since these systems are booting from USB, I´ve configured the systems to mount / as read-only to prolong their lives by quite far. What whould be the best-practice way to configure that in use with beadm? Up until now, I´ve only used plain-ol' fstab with all fs's legacy to handle things, so I´m quite new to ZFS´s automagic:)

First, do not use /etc/fstab for ZFS mounts, it will not work with beadm, use the zfs_enable=YES in /etc/rc.conf file.

Second, using plain file for ZFS becomes less relevant with every release, for example in 9.1-STABLE You no longer need to use the /boot/zfs/zpool.cache file and You do not need to set vfs.root.mountfrom in the /boot/loader.conf file anymore. Only bootfs property is used.

As for beadm to work with Boot Environment spread across several pool with the same prefix ... well, everything si possible, its about how much time You put into that, its definitely technically possible, but in its current implementation beadm does not support such schema and I am afraid I will not be adding these changes.

I will see what can be added to the beadm to support such schemas and will tell You if I will modify it to do that.
 
vermaden said:
The beadm operates on ${BOOTPOOL}/ROOT/${BENAME} and anything that is below that path is taken into the Boot Environment. Anything else, even with the same 'schema' is omitted ${OTHERPOOL}/ROOT/${BENAME} will not be snapshotted and used for beadm.
Yes, that´s what I was afraid of. Good to have that clarified.

vermaden said:
First, do not use /etc/fstab for ZFS mounts, it will not work with beadm, use the zfs_enable=YES in /etc/rc.conf file.
I had gathered as much.

vermaden said:
Second, using plain file for ZFS becomes less relevant with every release, for example in 9.1-STABLE You no longer need to use the /boot/zfs/zpool.cache file and You do not need to set vfs.root.mountfrom in the /boot/loader.conf file anymore. Only bootfs property is used.
Ah, sweet. Good to know, thanks!

vermaden said:
As for beadm to work with Boot Environment spread across several pool with the same prefix ... well, everything si possible, its about how much time You put into that, its definitely technically possible, but in its current implementation beadm does not support such schema and I am afraid I will not be adding these changes.

I will see what can be added to the beadm to support such schemas and will tell You if I will modify it to do that.
I totally understand not wanting to redo and possibly break what´s working. If you are ever bored enough and decide to go for it anyway, let me know:) Thanks again!

/Sebulon
 
This is one awesome thread. I said it before, I'll say it again: thanks :)

Recently I got notebook where I needed to use encryption. I want to have full encryption though. Basically it's what you've done but with a small modification.
I'm attaching it here, maybe it can be helpful.

Idea is to have encrypted the whole disk leaving only /boot partition unencrypted. Boot is be done from bootz ZFS pool, rpool is the encrypted root pool.

My final disk layout is as follows:

# gpart show ada0
Code:
=>       34  976773101  ada0  GPT  (465G)
         34          6        - free -  (3.0k) 
         40        256     1  freebsd-boot  (128k)         # ; bootcode
        296    4194304     2  freebsd-zfs  (2.0G)          # ; boot ZFS
    4194600  950009856     3  freebsd-zfs  (453G)          # ; encrypted ZFS
  954204456   22568672     4  freebsd-swap  (10G)          # ; dump 
  976773128          7        - free -  (3.5k)

# zpool status
Code:
  pool: bootz
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        bootz       ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          ada0p3.eli  ONLINE       0     0     0

errors: No known data errors

# zfs list
Code:
NAME                     USED  AVAIL  REFER  MOUNTPOINT
bootz                    350M  1.61G    31K  none
bootz/default            350M  1.61G   350M  /bootdir
rpool                    307M   445G   144K  none
rpool/ROOT               306M   445G   144K  none
rpool/ROOT/default       305M   445G  24.1M  legacy
rpool/ROOT/default/usr   281M   445G   281M  /usr
rpool/ROOT/default/var   868K   445G   868K  /var
rpool/ROOT/empty         144K   445G   144K  /var/empty
rpool/ROOT/home          184K   445G   184K  /home
rpool/ROOT/tmp           176K   445G   176K  /tmp

Where bootfs property is set for bootz/default. One has to deal with /boot to be set properly:

# cd /; ll boot
Code:
lrwxr-xr-x  1 root  wheel  12 Jan 26 00:20 boot@ -> bootdir/boot

This does screw with beadm I guess. It does require manual creation of bootz/* dataset and managing what is mounted to /bootdir.
 
matoatlantis said:
This is one awesome thread. I said it before, I'll say it again: thanks :)

Welcome.

matoatlantis said:
This does screw with beadm I guess. It does require manual creation of bootz/* dataset and managing what is mounted to /bootdir.

I have created a modified version of beadm for such setup, here is latest modified 0.8.4 version (0.8.5 unmodified in Ports):
http://forums.freebsd.org/showpost.php?p=195040

... at least it works with that setup:
http://forums.freebsd.org/showpost.php?p=195631
 
Hi, I try to build the Trois-Six configuration, but I have a problem I don't understand:

Code:
mounting from zfs:zroot/ROOT/default failed with error 2

Why?
 
@nORKy

You messed something up with zpool.cache (which is very easy brake something btw).

Fortunately 9.1-STABLE does not need zpool.cache 'hack' anymore.

To fix that You need to boot from live CD/USB and import the pool with -o cachefile-zpool.cache to 'regenerate' it.
 
vermaden said:
@nORKy

You messed something up with zpool.cache (which is very easy brake something btw).

Fortunately 9.1-STABLE does not need zpool.cache 'hack' anymore.

To fix that You need to boot from live CD/USB and import the pool with -o cachefile-zpool.cache to 'regenerate' it.

It works. Thank you.
 
If I read the detailed release-notes for FreeBSD 9.1 correctly it appears possible to install FreeBSD on ZFS without creating a seperate root-partition, see: http://www.freebsd.org/releases/9.1R/relnotes-detailed.html#boot Doing this to me makes sense for using beadm as everything then gets captured by beadm.

Unfortunately I can't get my head around how this change (as well as the zroot.cache one) affects the install-guide in the first post. So could someone please point me in the correct direction?

It seems to me that I should just follow the guide, while simply leaving out the creation of the boot-fs partition? If that is indeed the case, how do I deal with the zpool.cache change when installing FreeBSD 9.1?
 
@@moesasji

This guide *is* about creating ZFS on root without separate UFS boot partition. The GPT boot partition is mandatory to boot, there is no filesystem there, without it You will not boot, its 128 kilobytes in size by the way.

This guide was created when 9.1-RELEASE was available. There is small problem with 9.1-RELEASE that has already been fixed in STABLE: http://www.freebsd.org/cgi/query-pr.cgi?pr=167905 (You can of course add that patch and still use 9.1-RELEASE) This is needed change: http://freshbsd.org/commit/freebsd/r237119

About zpool.cache, its no longer needed (in STABLE/HEAD/CURRENT), the vfs.root.mountfrom at /boot/loader.conf is also not needed (in STABLE/HEAD/CURRENT), but that is not the case in 9.1-RELEASE.
 
Last edited by a moderator:
vermaden said:
@@moesasji
This guide *is* about creating ZFS on root without separate UFS boot partition. The GPT boot partition is mandatory to boot, there is no filesystem there, without it You will not boot, its 128 kilobytes in size by the way.

It is probably me misunderstanding the release notes for 9.1. Your (and other guides) installing FreeBSD on ZFS have a small freebsd-boot section created with
Code:
gpart add -t freebsd-boot -l bootcode${NUMBER} -s 128k ${I}

which I assume is the GPT boot partition you mention and is also the way how I installed it in the past. However if I look at the example given in the 9.1-changelog that line creating a freebsd-boot partition is no longer there. See last bit of the section: http://www.freebsd.org/releases/9.1R/relnotes-detailed.html#boot. I guess that is sparc64 specific, yet not sure why it would be different.

ps) thanks for the warning on the needed patch. Would have loved to be able to stick with a binary upgrade path. :-(
 
Last edited by a moderator:
@@moesasji

&quot said:
# gpart create -s vtoc8 da0
The VTOC8 partition schame is not GPT patition scheme, but maybe ZFS only booting without freebsd-boot partition is possible, I haven't tried ;)
 
Last edited by a moderator:
vermaden said:
@@moesasji
The VTOC8 partition schame is not GPT patition scheme, but maybe ZFS only booting without freebsd-boot partition is possible, I haven't tried ;)

After some digging it turns out I just have to learn to read the man-pages better. :r

The man-page of gpart states under bootstrapping that the GPT-scheme needs the freebsd-boot section to place /boot/gptzfsboot. The case for VTOC8 is indeed different as the man-page states:

The VTOC8 scheme does not support embedding bootstrap code. Instead, the 8 KBytes bootstrap code image /boot/boot1 should be written with gpart bootcode command with -p bootcode option to all sufficiently large VTOC8 partitions.

That explains a lot and hence no need to try. Sorry for the noise.
 
Last edited by a moderator:
Vermaden you may be interested in this.

*IMGUR* grub2 testing

Currently booting active and non-active BE's from the menu like how it is handled in Opensolaris/Illumos.

This bypasses loader, so all settings or modules loaded in your loader.conf would need to be set in grub.cfg, this will solve the issue of having to use a rescue media in the event you can't get a live system to rollback.

However, we will still have to wait for an updated grub2 in ports, the package I built is a test port thanks to nox-.
 
@@dkeav

I have talked with FreeBSD developers about that as a possible solution, but we all agreed, that using GNU code for that is questionable and shortcoming. We agreed that they will start to update/modify the boot code to create that menu before the ZFS loader, to have native solution. There were several commits but its nowhere near complete.

You can use GRUB2 of course, but You will need some modern Linux install for that, because as You mentioned, GRUB2 is Ports is in 1.98 version while booting from ZFS has been added at 1.99 and 1.99 or later is not in Ports beucase of older binutils from what I recall.

Making beadm to put valid entries into grub.cfg or menu.lst is very easy, but as we agreed that there would be native solution and that GRUB2 will not make fast into the Ports I did not added it.

BTW, about Your screenshots, at 9.1-STABLE there is no more need for zpool.cache and You do not need to set vfs.root.mountfrom as zpool property bootfs is used now. It works beatifully with/without beadm but I havent tried it with GRUB2.
 
Last edited by a moderator:
I knew about zpool.cache going away, but testing is being done on 9.1-RELEASE. I'll switch to bootfs after I get things working with UEFI.
 
I installed beadm on my FreeBSD 9.1 i386 system that has an encrypted ZFS root but I get an error when I run the beadm list command:
Code:
ERROR: This system is not configured for boot environments

Can this utility be used on an existing system? Or does a new system install need to be configured in such a way that beadm will work?
 
I was able to use it on my NAS system that wasn't prepared with beadm. All I had to do was to rename the datasets to conform to the poolname/ROOT/default naming scheme.
 
Back
Top