ZFS exclude zfs pool from boot

robot468 · Jul 19, 2022

man for gptzfsbool says "The first pool seen during probing is used as a default boot pool."
Is there any way to boot from another pool?

gpw928 · Jul 20, 2022

robot468 said:
man for gptzfsbool says "The first pool seen during probing is used as a default boot pool."
Is there any way to boot from another pool?

Here is an extract of the plan I used to move the root of my ZFS server form pool "zroot" (the default pool name) to the new pool "zroot2". This boots from a pool on new disks, which is the most general case. I have not shown the partitioning required for the new disks, but that's pretty obvious (swap on partition 2, root pool on partition 3). The part you really need is the "zpool set bootfs", but the surrounding context is important.

Code:

DISK0=ada2      # first disk of new root mirror
DISK1=ada3      # second disk of new root mirror
SWAP=swap2      # new swap GEOM mirror
ZROOTSRC=zroot  # source pool (old root)
ZROOTDST=zroot2 # destination pool (new root)

# Create new swap and root
gmirror label -v -b round-robin $SWAP /dev/${DISK0}p2
gmirror insert $SWAP /dev/${DISK1}p2
zpool create $ZROOTDST mirror /dev/${DISK0}p3 /dev/${DISK1}p3

# Copy the old root to the new root.
zfs snapshot -r $ZROOTSRC@replica1
zfs umount $ZROOTDST    # you must keep it unmounted
zfs send -R $ZROOTSRC@replica1 | zfs receive -Fdu $ZROOTDST

# The new root is now frozen at the time of the snapshot.
# If this is an issue you need to drop into single user mode
# to execute the snapshot prior to send/receive.

# This is the default bootable dataset for the new root pool.
# It's usually <zroot_pool_name>/ROOT/default.
# But an upgrade using a different boot environment may change that.
# You must get this right, or your system will not boot.
# Run "zpool get bootfs $ZROOTSRC" to see the value for the old root.
# Mine looks like "zroot/ROOT/13".  Yours may be "zroot/ROOT/default".
# Keep the "ROOT/13" or "root/default" part and change the pool name
# ZROOTSRC to ZROOTDST (i.e. "zroot" to "zroot2").
zpool set bootfs=$ZROOTDST/ROOT/13 $ZROOTDST
zpool export $ZROOTDST

# Reboot, but interrupt it to re-confgiure the BIOS.
# Edit the BIOS boot order to favour new root mirrors, e.g. ada2, ada3.
# Reset, and allow the system to boot SINGLE USER mode.
# We need to stop the old zroot from being imported and mounted.
# https://openzfs.github.io/openzfs-docs/Project%20and%20Community/\
#         FAQ.html#the-etc-zfs-zpool-cache-file
# Execute these commands in single user mode.
zfs set readonly=off $ZROOTDST
rm -f /boot/zfs/zpool.cache /etc/zfs/zpool.cache
zpool set cachefile=/etc/zfs/zpool.cache $ZROOTDST
# Change fstab to use the new swap partition.  I'm using a GEOM mirror.
# fstab: /dev/mirror/$SWAP none swap sw 0 0
vi /etc/fstab

gpw928 · Jul 20, 2022

gpw928 said:
I have not shown the partitioning required for the new disks, but that's pretty obvious (swap on partition 2, root pool on partition 3).

For the sake of completeness, here is the code to partition a fresh set of two SSDs for a new ZFS root mirror:

Code:

# create the partition tables
gpart destroy -F ${DISK0}
gpart destroy -F ${DISK1}
gpart create -s GPT ${DISK0}
gpart create -s GPT ${DISK1}

# Create a 512 kB boot partition at offset 40 -- which is the size of the
# FAT_32 "reserved sectors" (32 or 34 blocks) rounded up to 4 kB boundary.
# This is the same layout used by the FreeBSD 13 intaller.
gpart add -i 1 -b 40 -s 512k -t freebsd-boot ${DISK0}
gpart add -i 1 -b 40 -s 512k -t freebsd-boot ${DISK1}

# Install the first and second stage bootloaders for a ZFS root
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${DISK0}
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${DISK1}

# Allign all subsequent partitions on a 1 MiB boundary
gpart add -a 1m -i 2 -s 16g -t freebsd-swap ${DISK0}    # swap (GEOM mirror)
gpart add -a 1m -i 2 -s 16g -t freebsd-swap ${DISK1}
gpart add -a 1m -i 3 -s 86g -t freebsd-zfs ${DISK0}     # root (ZFS mirror)
gpart add -a 1m -i 3 -s 86g -t freebsd-zfs ${DISK1}
gpart add -a 1m -i 4 -s 12g -t freebsd-zfs ${DISK0}     # ZIL (ZFS mirror)
gpart add -a 1m -i 4 -s 12g -t freebsd-zfs ${DISK1}
gpart add -a 1m -i 5 -s 64g -t freebsd-zfs ${DISK0}     # L2ARC (ZFS stripe)
gpart add -a 1m -i 5 -s 64g -t freebsd-zfs ${DISK1}
# The rest is TRIM'd (newfs -E) and unused (overprovisioning on the SSD)
gpart add -a 1m -i 6 -t freebsd-ufs ${DISK0}            # unused
gpart add -a 1m -i 6 -t freebsd-ufs ${DISK1}

T-Daemon · Jul 20, 2022

robot468, you mention gptzfsboot(8), therefore I assume the hardware is a BIOS-based system.

If the pools are on different disks and the pool which should be booted from now on is on a disk which has a gptzfsboot(8) bootstrap code partition of its own (freebsd-boot) just change the boot order in BIOS.

If the case is the pools are on the same disk and all ZFS pools shall be keept and not moved on an other disk, apply configuration down below. The procedure was taken from the gptzfsboot(8) manuals USAGE section.

To configure a new default ZFS boot pool, create in the first pools file system gptzfsboot(8) detects (the current default boot pool), file /boot.config and set

Code:

zfs:xxx/ROOT/default:

Replace "xxx" with pool name, and mind the colon at the end of the file system specification.

To exclude a unused zfs pool (or pools) from boot permanently, its zpoolprops(7) "bootfs" can be unset.

robot468 · Jul 20, 2022

gpw928 said:
Edit the BIOS boot order to favour new root mirrors, e.g. ada2, ada3.

My crossflashed H710P does not support selecting a drive to boot. It just tries all the disks one by one.

I thought that in order to exclude an unnecessary pool from booting, I could change its partition type with "gpart modify" from freebsd-zfs to something else. The man on gptzfsboot says that it only tries freebsd-zfs partitions.

What do you think about this?

T-Daemon said:
To configure a new default ZFS boot pool, create in the first pools file system gptzfsboot(8) detects (the current default boot pool), file /boot.config and set

I was just interested in this method, but the man about it does not say too much detail. It looks like the most painless solution, I'll give it a try, thanks.

T-Daemon said:
To exclude a unused zfs pool (or pools) from boot permanently, its zpoolprops(7) "bootfs" can be unset.

Man gptzfsboot says: "If the bootfs property is not set, then the root filesystem of the pool is used as the default."

Andriy · Jul 20, 2022

If you feel comfortable rebuilding and reinstalling gptzfsboot, then you can try a patch from here https://reviews.freebsd.org/D33302

T-Daemon · Jul 20, 2022

robot468 said:
I thought that in order to exclude an unnecessary pool from booting, I could change its partition type with "gpart modify" from freebsd-zfs to something else. The man on gptzfsboot says that it only tries freebsd-zfs partitions.

What do you think about this?

I considered that option as well, but that doesn't work. The boot process stops at the BTX "boot" loader prompt.

robot468 said:
Man gptzfsboot says: "If the bootfs property is not set, then the root filesystem of the pool is used as the default."

I believe that sentence is misphrased. If there are multiple ZFS pools, gptzfsboot(7) is booting the first pool it finds from a freebsd-zfs partition with the ZFS pool property "bootfs" set.

If a pool has "bootfs" unset, the next pool in line having it set is booted. I verified by testing in a VM.

T-Daemon · Jul 20, 2022

Andriy said:
If you feel comfortable rebuilding and reinstalling gptzfsboot, then you can try a patch from here https://reviews.freebsd.org/D33302

The reviews summary says

zfs boot: prefer a pool with bootfs set for a boot pool

In other words, ignore pools without bootfs property when considering
candidates for a bool pool. Use a pool without bootfs only if none of
discovered pools has it.

This should allow to easily designate a boot pool on a system with
multiple pools without having to shuffle disks, change firmware boot
order, etc.

I speak here for 13.1-RELEASE, isn't that the case already, except "Use a pool without bootfs only if none of discovered pools has it.". See explanation post #7.

robot468 · Jul 20, 2022

Andriy said:
If you feel comfortable rebuilding and reinstalling gptzfsboot, then you can try a patch from here https://reviews.freebsd.org/D33302

This diff is applicable to any particular version of the source code, right? How do I know that I'm applying it to the correct one?

Andriy · Jul 20, 2022

T-Daemon I believe that you had some problem in your testing. gptzfsboot manual has the correct wording. bootfs property does not play any role in selecting from which pool to boot, bootfs only plays role in selecting from which filesystem to boot on the chosen pool.

Andriy · Jul 20, 2022

robot468 said:
This diff is applicable to any particular version of the source code, right? How do I know that I'm applying it to the correct one?

If the diff applies cleanly, then it is applicable; if not, then not. Hope that helps

robot468 · Jul 20, 2022

T-Daemon said:
If a pool has "bootfs" unset, the next pool in line having it set is booted. I verified by testing in a VM.

If so, that solves all the problems. I will try it now on the VM, thank you

Andriy · Jul 20, 2022

T-Daemon said:
The reviews summary says

I speak here for 13.1-RELEASE, isn't that the case already, except "Use a pool without bootfs only if none of discovered pools has it.". See explanation post #7.

As I said in the other comment, I believe that it is not the case.

T-Daemon · Jul 20, 2022

Andriy said:
T-Daemon I believe that you had some problem in your testing. gptzfsboot manual has the correct wording. bootfs property does not play any role in selecting from which pool to boot, bootfs only plays role in selecting from which filesystem to boot on the chosen pool.

I'll check again and get back.

T-Daemon · Jul 20, 2022

Andriy said:
I believe that you had some problem in your testing.

You are right. I can't reproduce it anymore.

I'm not sure what I did then, I though after un-setting "bootfs" on the first pool the system booted from the next pool. That is not the case, it drops to the BTX boot prompt, after searching and not finding a loader or kernel.

robot468, I'm sorry to have put you on the wrong track.

Thanks for explaining and clarifying the issue Andriy.

T-Daemon · Jul 26, 2022

Hi Andriy,

I've tested your patch on 13.1-RELEASE, but it didn't work as expected.

My test system is a VirtualBox VM (BIOS and EFI), with two disks, on each a FreeBSD installation, zroot0 (disk 0) and zroot1 (disk 1):

- Boot zroot1 (BIOS menu)

Code:

# zpool import -R /mnt zroot0
# zpool set bootfs=   zroot0
# zpool get bootfs
NAME    PROPERTY  VALUE                SOURCE
zroot0  bootfs    -                    default
zroot1  bootfs    zroot1/ROOT/default  local

- Reboot system

When BIOS (transcript):

Code:

BIOS drive C: is disk0
BIOS drive D: is disk1

Can't find /boot/zfsloader
Can't find /boot/loader
Can't find /boot/kernel/kernel

FreeBSD/x86 boot
Default: zfs:zroot0:/boot/kernel/kernel
boot:

Can't find /boot/kernel/kernel

FreeBSD/x86 boot
Default: zfs:zroot0:/boot/kernel/kernel
boot:

When EFI:

The patch does apply correctly, no errors shown when patched. The file size of the patched gptzfsboot is greater than the unpatched version and consequently the checksum is different .

Also diff(1) of zfsimpl.c and zfsiml.h looks ok.

Code:

# diff 13.1-RELEASE/stand/libsa/zfs/zfsimpl.c     13.1-RELEASE-bootfs/stand/libsa/zfs/zfsimpl.c

1344a1345,1359
> spa_first_bootable(void)
> {
> spa_t *spa;
>
> /* Pick the first pool with "bootfs" property explicitly set, if any. */
> STAILQ_FOREACH(spa, &zfs_pools, spa_link) {
> if (spa->spa_bootfs != 0)
> return (spa);
> }
>
> /* Fall back to the first pool if there is one. */
> return (STAILQ_FIRST(&zfs_pools));
> }
>
> static spa_t *
1376c1391
< return (STAILQ_FIRST(&zfs_pools));
---
> return (spa_first_bootable());
3296,3297c3311,3312
< dnode_phys_t dir, propdir;
< uint64_t props, bootfs, root;
---
> dnode_phys_t dir;
> uint64_t root;
3301,3307c3316,3319
< /*
< * Start with the MOS directory object.
< */
< if (objset_get_dnode(spa, spa->spa_mos,
< DMU_POOL_DIRECTORY_OBJECT, &dir)) {
< printf("ZFS: can't read MOS object directory\n");
< return (EIO);
---
> /* See if the pool has bootfs set. */
> if (spa->spa_bootfs != 0) {
> *objid = spa->spa_bootfs;
> return (0);
3311,3321d3322
< * Lookup the pool_props and see if we can find a bootfs.
< */
< if (zap_lookup(spa, &dir, DMU_POOL_PROPS,
< sizeof(props), 1, &props) == 0 &&
< objset_get_dnode(spa, spa->spa_mos, props, &propdir) == 0 &&
< zap_lookup(spa, &propdir, "bootfs",
< sizeof(bootfs), 1, &bootfs) == 0 && bootfs != 0) {
< *objid = bootfs;
< return (0);
< }
< /*
3323a3325,3330
> if (objset_get_dnode(spa, spa->spa_mos, DMU_POOL_DIRECTORY_OBJECT,
> &dir)) {
> printf("ZFS: failed to read pool %s directory object\n",
> spa->spa_name);
> return (EIO);
> }
3325,3326c3332
< sizeof(root), 1, &root) ||
< objset_get_dnode(spa, spa->spa_mos, root, &dir)) {
---
> sizeof(root), 1, &root) != 0) {
3329a3336,3339
> if (objset_get_dnode(spa, spa->spa_mos, root, &dir) != 0) {
> printf("ZFS: can't read root dsl_dir\n");
> return (EIO);
> }
3473,3474c3483,3484
< dnode_phys_t dir;
< uint64_t config_object;
---
> dnode_phys_t dir, propdir;
> uint64_t config_object, props;
3501a3512,3527
> }
>
> /*
> * Lookup the pool_props and see if we can find a bootfs.
> */
> if (zap_lookup(spa, &dir, DMU_POOL_PROPS,
> sizeof(props), 1, &props) == 0) {
> rc = objset_get_dnode(spa, spa->spa_mos, props, &propdir);
> if (rc != 0) {
> printf("ZFS: failed to read pool %s properties object\n",
> spa->spa_name);
> return (rc);
> }
> spa->spa_bootfs = 0;
> (void)zap_lookup(spa, &propdir, "bootfs",
> sizeof(spa->spa_bootfs), 1, &spa->spa_bootfs);

Code:

# diff 13.1-RELEASE/sys/cddl/boot/zfs/zfsimpl.h     13.1-RELEASE-bootfs/sys/cddl/boot/zfs/zfsimpl.h
1871a1872
>     uint64_t    spa_bootfs;    /* bootfs object id */

SirDice · Jul 26, 2022

gptboot(8) checks the bootme and bootonce attributes:

Code:

     bootme          Attempt to boot from this partition.  If more than one
                     partition has the bootme attribute set, gptboot will
                     attempt to boot each one until successful.

     bootonce        Attempt to boot from this partition only one time.
                     Setting this attribute with gpart(8) automatically also
                     sets the bootme attribute.  Multiple partitions may have
                     the bootonce and bootme attributes set.

gptzfsboot(8) seems to completely ignore those. Maybe it can be implemented for gptzfsboot(8)? That would make things more consistent and you get control over which freebsd-zfs to boot from.

Andriy · Jul 26, 2022

T-Daemon what you see is what I would expect from the unpatched gptzfsboot...
Did you actually install it into the boot partition?
Do you have boot partitions on both disks or just on one of them?

Also, I think that EFI boot does not use gptzfsboot. I think that it uses efiloader that's installed in the efi / msdos filesystem.

T-Daemon · Jul 29, 2022

Andriy said:
Did you actually install it into the boot partition?

After applying the patch on 13.1-RELEASE and building world, I didn't install gptzfsboot into a existing boot partition manually.

A installer image was created (buildworld buildkernel, make disc1.iso) and used to install the systems on the two disk VM mentioned earlier.

Andriy said:
Do you have boot partitions on both disks or just on one of them?

On both.

Code:

=>     40  4194224  ada0  GPT  (2.0G)
       40     1024     1  freebsd-boot  (512K)
     1064      984        - free -  (492K)
     2048   409600     2  freebsd-swap  (200M)
   411648  3780608     3  freebsd-zfs  (1.8G)
  4192256     2008        - free -  (1.0M)

=>     40  4194224  ada1  GPT  (2.0G)
       40     1024     1  freebsd-boot  (512K)
     1064      984        - free -  (492K)
     2048   409600     2  freebsd-swap  (200M)
   411648  3780608     3  freebsd-zfs  (1.8G)
  4192256     2008        - free -  (1.0M)

The patch was applied correctly [1], and the compiled and installed gptzfsboot in freebsd-boot is the one patched [2].

[1] diff(1) between original and patched files show applied differences.

[2] on both installed systems in the VM, strings(1) /dev/ada0p1|ada1p1 shows following included message, absent in the original zfsimpl.c: "ZFS: failed to read pool %s properties object"

From patch

Code:

@@ -3488,6 +3498,22 @@
...
+               printf("ZFS: failed to read pool %s properties object\n",

Andriy said:
Also, I think that EFI boot does not use gptzfsboot. I think that it uses efiloader that's installed in the efi / msdos filesystem.

Of course! I had a misconception there.

Andriy · Jul 29, 2022

T-Daemon could you please check if both disks are visible to the boot blocks?
I think that you can enter status at the prompt to see available pools.
You can also enter zfs:zroot0/ROOT/default:/boot/zfsloader to boot to the loader and there you can use lsdev (lsdev -v).

Andriy · Jul 29, 2022

Actually, I found a problem with my change. zfsboot code has an additional check for the "BIOS" boot disk ID.
I'm working on an update patch.

T-Daemon · Jul 30, 2022

Andriy said:
[FONT=monospace]T-Daemon[/FONT] could you please check if both disks are visible to the boot blocks?

Just in case it's still of interest.

Andriy · Aug 1, 2022

T-Daemon I have update the patch in the review.

Also, I looked at the EFI boot code and it seems like it tries only partitions on the boot disk (the disk on which the ESP / EFI partition was used for booting). It does not consider other disks. That can also be seen in your screenshot in the earlier comment , partitions 4, 1, 2, 3 from the same disk were tried. I may try to work on that code too, but no promises.

T-Daemon · Aug 2, 2022

Andriy said:
I have update the patch in the review.

Thank you for your effort, now it works as expected.

I'v tested on a multi disk system (BIOS+UEFI). Each has the installer images default, efi loader, bootcode, swap, ZFS pool partitions, and a 1 disk system with one efi, one bootcode, freebsd-swap, two freebsd-zfs partitions.

In BIOS and UEFI, pools with the "bootfs" unset are skipped and the next pool in line with the flag set is booted.

Andriy said:
Also, I looked at the EFI boot code and it seems like it tries only partitions on the boot disk (the disk on which the ESP / EFI partition was used for booting). It does not consider other disks. I may try to work on that code too, but no promises.

That my not be necessary. This is what I observed on UEFI:

On a 4 disk VirtualBox VM setup, consisting of each disk with efi, freebsd-boot, freebsd-swap, freebsd-zfs partitions, I've unset the first 3 disks pools "bootfs" flag.

When the efi loader (on disk0) fails to find a bootable partition on "his" disk, it begins to count down for reboot.

After the countdown ends it doesn't reboot there. If I interpreted the screenshots down below correctly, the next loader (on disk1) begins to probes "his" disk, When the loader doesn't find a bootable partition there, the next loader (on disk2) probes "his" disk.

In the above scenario, three times the loader has count down for reboot, the last, 4th pool, with the "bootfs" flag set, was booted.

Efi loader probing disk0:

Efi loader probing disk1:

Efi loader probing disk2:

Andriy · Aug 2, 2022

That's very interesting. I am not sure if this is the EFI mandated behavior or how a particular implementation does things.

But there is a tricky bit too.
loader.efi checks if selected ZFS filesystem has some mandatory files and skips the pool if it doesn't.
So, when bootfs is unset on a pool, then by default the pool's root dataset would be checked. If it's just empty or has only unrelated files, then it would be skipped.
But if it has a FreeBSD installation then loader.efi would proceed to boot from it.

So, your test worked almost by accident. That is, it worked not because bootfs was unset, but because the default value of bootfs pointed to a filesystem without FreeBSD installed.
I guess that the difference is not significant from the practical point of view, just wanted to point it out.

ZFS exclude zfs pool from boot

Administrator