Solved How to exclude some datasets from a boot environment

vienuolis · Tuesday at 3:23 PM

I am relatively novice, with no Unix experience. I manage my barebone server for our small community, mainly for Web publishing (Textpattern), email (OpenSMTPD with Cyrus IMAP), and storage (Nextcloud) on FreeBSD 14.3-R. I do not have a spare computer for testing. After ZFS rollback disaster I carefully read the handbook, wiki, manuals, Dan Langille', Slawomir Wojciech' (vermaden), Clara's tips on ZFS and BE, but have not found my mistake and I still lack clarity, so I want to ask you before cleaning and activating the new BE. The default “shallow” boot environment structure include /usr, /var, and exclude /var/mail:

Code:

> zfs list -o name,canmount,mountpoint
NAME                                          CANMOUNT  MOUNTPOINT
zroot                                         on        /zroot
zroot/ROOT                                    on        none
zroot/ROOT/13.0-RELEASE_2022-03-09_default    noauto    /
[…]
zroot/ROOT/14.2-Rp3                           noauto    /
zroot/ROOT/14.3-RELEASE_2025-06-09_171055     noauto    /
zroot/ROOT/14.3-RELEASE_2025-07-13_113047     noauto    /
zroot/ROOT/14.3p1_2025-07-13boot              noauto    /
zroot/ROOT/14.3p1_2025-07-25boot              noauto    /
zroot/ROOT/14.3-RELEASE-p1_2025-08-08_172418  noauto    /
zroot/ROOT/14.3-RELEASE-p2_2025-10-03_153256  noauto    /
zroot/tmp                                     on        /tmp
zroot/usr                                     off       /usr
zroot/usr/home                                on        /usr/home
zroot/usr/ports                               on        /usr/ports
zroot/usr/src                                 on        /usr/src
zroot/var                                     off       /var
zroot/var/audit                               on        /var/audit
zroot/var/crash                               on        /var/crash
zroot/var/log                                 on        /var/log
zroot/var/mail                                on        /var/mail
zroot/var/tmp                                 on        /var/tmp

Cyrus IMAP stores email in /var/spool/cyrus, so I intend to exclude it (and maybe also /var/db/mysql, /usr/local/www/nextcloud-files) from BE, but I am not sure how:

Code:

> zfs set canmount=on zroot/var/spool/cyrus
cannot open 'zroot/var/spool/cyrus': dataset does not exist
> zfs create -nv zroot/var/spool/cyrus
would create zroot/var/spool/cyrus

Is this the right way to go?

SirDice · Tuesday at 4:09 PM

Yes, essentially zroot/var is a place holder, it's not mounted so it's actually part of the BE. That place holder just makes it easier to create zroot/var/mail for example and have it 'automagically' assume it should have /var/mail/ as its mountpoint.

Now, /var/spool is not a dataset, but you might want to keep everything else under /var/spool (like /var/spool/lock) included in the BE. So I'd do something like this:

Code:

zfs create -o canmount=off zroot/var/spool
zfs create -o canmount=on zroot/var/spool/cyrus

You need to have a zroot/var/spool or else you cannot create zroot/var/spool/cyrus. This is similar to files and directories, you cannot create /tmp/somedir/somefile if the /tmp/somedir directory doesn't exist.

Because zroot/var has the mountpoint /var/, zroot/var/spool will automatically assume /var/spool mountpoint, and zroot/var/spool/cyrus will become /var/spool/cyrus.

You could also do something completely different, and specifically split off data from the OS/applications:

Code:

zfs create -o canmount=off -o mountpoint=none zroot/DATA
zfs create -o canmount=on -o mountpoint=/var/db/mysql zroot/DATA/mysql
zfs create -o canmount=on -o mountpoint=/var/spool/cyrus zroot/DATA/cyrus

With regards to cleaning up your current BE mess, look at bectl list, remove/destroy ( bectl destroy {...}) everything that's not marked "NR" ("N" is active Now; "R" will become activate after a Reboot)

Erichans · Tuesday at 4:13 PM

Yes, this resembles somewhat how physical partitions are mapped into the hierarchy on a traditional fs.

You may find the presentation by Allan Jude (ZFS Powered Magic Upgrades) helpful where he speaks about including Xorg etc. in a BE on his laptop, but not on his server(s): overview of various more info about BE's: Boot Environment & snapshot - important ZFS features.

vienuolis · Tuesday at 8:27 PM

Ouch! I followed your second solution, SirDice — very reasonable to split off user data from the OS/applications:

Code:

> zfs list
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
zroot                                          206G  1.48T    96K  /zroot
zroot/DATA                                     384K  1.48T    96K  none
zroot/DATA/cloud                                96K  1.48T    96K  /usr/local/www/duo
zroot/DATA/cyrus                                96K  1.48T    96K  /var/spool/cyrus
zroot/DATA/mysql                                96K  1.48T    96K  /var/db/mysql
zroot/ROOT                                     175G  1.48T    96K  none
zroot/ROOT/13.0-RELEASE_2022-03-09_default    22.7M  1.48T  45.8G  /
[…]
zroot/ROOT/14.3-RELEASE-p2_2025-10-03_153256     8K  1.48T  49.8G  /
zroot/tmp                                     19.4M  1.48T  19.4M  /tmp
zroot/usr                                     29.0G  1.48T    96K  /usr
zroot/usr/home                                29.0G  1.48T  27.4G  /usr/home
zroot/usr/ports                                 96K  1.48T    96K  /usr/ports
zroot/usr/src                                   96K  1.48T    96K  /usr/src
zroot/var                                      675M  1.48T    96K  /var
zroot/var/audit                                 96K  1.48T    96K  /var/audit
zroot/var/crash                                 96K  1.48T    96K  /var/crash
zroot/var/log                                  674M  1.48T   674M  /var/log
zroot/var/mail                                1004K  1.48T  1004K  /var/mail
zroot/var/tmp                                  164K  1.48T   164K  /var/tmp

But now the mail has become inaccessible, even though Cyrus IMAP is still working and no warnings in /var/log/maillog.

Code:

> smtpctl show status
MDA running
MTA running
SMTP running
> service imapd status
cyrus_imapd is running as pid 69106.

Nextcloud files also became inaccessible: Your data directory is invalid. Although websites via MySQL are operational. Should I change anything in imapd.conf, or somehow restore /var/spool/cyrus back?

Erichans · Tuesday at 9:05 PM

vienuolis said:
But now the mail has become inaccessible,

I suspect that the data is still there, but hidden by your new mount mapping ...
You've created new ZFS fs's (in ZFS: datasets): did you ~~copy~~move the corresponding data to the new dataset before the new mount became active?

To compare this to the situation in an environment of a normal UFS fs, think of a newly created and UFS formatted partition, ready to be mounted, for example, intended for a current user X's home dir. If you don't move X's existing data to that new partition before activating the mount, that data will be inaccessible while the partition is mounted.

SirDice · Tuesday at 10:05 PM

Erichans said:
I suspect that the data is still there, but hidden by your new mount mapping ...

Yep, that's very likely the reason.

vermaden · Tuesday at 10:22 PM

@vienuolis

The idea behind this canmont=off for /var and /usr goes like that:

The /var and /usr datasets data/files are in the BE - in the zroot/ROOT/default BE that is created at installer - and the rest like these:

zroot/tmp
zroot/usr/home
zroot/usr/ports
zroot/usr/src
zroot/var/audit
zroot/var/crash
zroot/var/log
zroot/var/mail
zroot/var/tmp

Are EXCLUDED from BE.

So by default everything is included in BE - and if you want to EXCLUDE some parts of /usr or /var - you add needed ZFS datasets for these EXCLUDES.

Hope that helps.

vienuolis · Tuesday at 10:29 PM

Sorry, what should I do now?

SirDice · Tuesday at 10:57 PM

I should perhaps have stated the obvious, well, obvious to anyone with at least some sysadmin experience. When you mess around with filesystems and mounts, make sure to STOP the service that's using the filesystem and/or mount.

When you zfs create {....} -o mount=/somewhere/here {....} myzpool/mydataset a new, and therefor empty, filesystem will get immediately mounted on /somewhere/here. If there are any existing files and/or directories underneath /somewhere/here, they will get 'masked' or 'hidden' underneath the newly mounted ZFS dataset.

How to fix this? Basic strategy, zfs umount /usr/local/www/duo, now your 'original' content of /usr/local/www/duo will be available again. Move it out of the way; mv /usr/local/www/duo /usr/local/www/duo.orig for example, recreate the directory mkdir /usr/local/www/duo. The directory has to exist for the mountpoint to work, then zfs mount /usr/local/www/duo. Now copy everything from /usr/local/www/duo.orig to /usr/local/www/duo. Then start your service again.

vienuolis · Wednesday at 2:29 AM

My HTTP server is Hiawatha, so

Code:

# service hiawatha stop
> service hiawatha status
hiawatha is not running.
# zfs umount /usr/local/www/duo
# mv /usr/local/www/duo /usr/local/www/duomuo
# mkdir /usr/local/www/duo
# zfs mount /usr/local/www/duo
cannot open '/usr/local/www/duo': leading slash in name
# zfs mount zroot/usr/local/www/duo
cannot open 'zroot/usr/local/www/duo': dataset does not exist
# zfs mount
zroot/ROOT/14.3p1_2025-07-25boot  /
zroot                           /zroot
zroot/var/mail                  /var/mail
zroot/tmp                       /tmp
zroot/usr/home                  /usr/home
zroot/usr/src                   /usr/src
zroot/usr/ports                 /usr/ports
zroot/var/audit                 /var/audit
zroot/var/crash                 /var/crash
zroot/var/log                   /var/log
zroot/var/tmp                   /var/tmp
zroot/DATA/mysql                /var/db/mysql
zroot/DATA/cyrus                /var/spool/cyrus
# zpool history | grep zroot/DATA
2025-10-28.21:33:56 zfs create -o canmount=off -o mountpoint=none zroot/DATA
2025-10-28.21:34:57 zfs create -o canmount=on -o mountpoint=/var/db/mysql zroot/DATA/mysql
2025-10-28.21:35:41 zfs create -o canmount=on -o mountpoint=/var/spool/cyrus zroot/DATA/cyrus
2025-10-28.21:37:14 zfs create -o canmount=on -o mountpoint=/usr/local/www/duo zroot/DATA/cloud
> ls /usr/local/www
duo
duomuo
hiawatha
nextcloud

Sorry, I really have no sysadmin experience.

vienuolis · Wednesday at 4:13 AM

I copied Nextcloud user assets from duomuo to duo despite duo failed to mount. The ZFS structure now looks pretty normal:

Code:

# zfs list -o name,canmount,mountpoint
NAME                                          CANMOUNT  MOUNTPOINT
zroot                                         on        /zroot
zroot/DATA                                    off       none
zroot/DATA/cloud                              on        /usr/local/www/duo
zroot/DATA/cyrus                              on        /var/spool/cyrus
zroot/DATA/mysql                              on        /var/db/mysql
zroot/ROOT                                    on        none
zroot/ROOT/13.0-RELEASE_2022-03-09_default    noauto    /
[…]
zroot/ROOT/14.3-RELEASE-p2_2025-10-03_153256  noauto    /
zroot/tmp                                     on        /tmp
zroot/usr                                     off       /usr
zroot/usr/home                                on        /usr/home
zroot/usr/ports                               on        /usr/ports
zroot/usr/src                                 on        /usr/src
zroot/var                                     off       /var
zroot/var/audit                               on        /var/audit
zroot/var/crash                               on        /var/crash
zroot/var/log                                 on        /var/log
zroot/var/mail                                on        /var/mail
zroot/var/tmp                                 on        /var/tmp

I can not explain how, but Nextcloud is working well again.

vienuolis · Wednesday at 5:00 AM

My crazy guess turned out to be right: after

zfs create -o canmount=on -o mountpoint=/var/spool/cyrus zroot/DATA/cyrus

the only command is sufficient:

zfs umount /var/spool/cyrus.

Now Cyrus IMAP is working too, hooray!

I left the same thing to do with the MySQL database, I am still afraid while our websites are fully operational — perhaps until service mysql-server restart?

vienuolis · Wednesday at 2:41 PM

Finally, I unmounted the MySQL databases, and now all three user datasets are separated from zroot/ROOT to the common zroot/DATA array.

Many thanks to SirDice, Erichans, and vermaden for clarification, especially for the principle of purpose separation in the pool. It would be nice to document this tip, even implement the principle as default in FreeBSD. I would see three or four destination parts in a single pool:

ROOT — for operating system and applied software,
DATA — common storage (databases, email),
HOME — personal user assets,
TEMP — temporary data, non-preserved on reboot.

While learning, I even missed one term in ZFS glossary:
pool (zroot) » tank? (ROOT) » dataset » subsets.

Now I am going to clean up, make backups, snaps, and prepare for the current FreeBSD upgrade. See you later!

vienuolis · Wednesday at 6:19 PM

Disaster — all zroot/DATA assets are lost.

MySQL data, storage files, all WWW, entire mail… everywhere is empty after reboot.

How can that be possible?

vienuolis · Thursday at 4:10 AM

What kind of beast is DATA? A non-existent ghost that destroys everything associated with it. What does it mean physically? Why is it undocumented?

vienuolis · Thursday at 6:59 AM

These zroot/DATA datasets in question are mounted — and yet wiped away straight after reboot:

Code:

> zfs get mounted,mountpoint zroot/DATA/cyrus
NAME              PROPERTY    VALUE             SOURCE
zroot/DATA/cyrus  mounted     yes               -
zroot/DATA/cyrus  mountpoint  /var/spool/cyrus  local

> zfs get mounted,mountpoint zroot/var/spool/cyrus
cannot open 'zroot/var/spool/cyrus': dataset does not exist

vienuolis · Thursday at 9:15 AM

I successfully revived services and recovored all the lost data while preserving the zroot/DATA partition. Although it remains unclear to me why reboot caused this disaster, so I am still trying to figure out how to avoid such accident in the future.

vienuolis · Friday at 7:13 AM

I am a little unsure and cannot find clarity in ZFS docs.

SirDice said:
zfs create -o canmount=off -o mountpoint=none zroot/DATA
zfs create -o canmount=on -o mountpoint=/var/db/mysql zroot/DATA/mysql

If DATA stands as a placeholder for MySQL and similar datasets, should not it be marked canmount=on, as ROOT is?

Code:

> zfs list -o name,canmount,mounted,mountpoint
NAME                                  CANMOUNT  MOUNTED  MOUNTPOINT
zroot                                 on        yes      /zroot
zroot/DATA                            off       no       none
zroot/DATA/cloud                      on        yes      /usr/local/www/duo
zroot/DATA/cyrus                      on        yes      /var/spool/cyrus
zroot/DATA/mysql                      on        yes      /var/db/mysql
zroot/ROOT                            on        no       none
zroot/ROOT/14.3-R5p_2025-10-30boot    noauto    yes      /
zroot/ROOT/14.3-R5p_2025-10-30good    noauto    no       /
zroot/ROOT/14.3-R4p_2025-10-29        noauto    no       /
zroot/ROOT/14.3-R1p_2025-07-25        noauto    no       /
zroot/tmp                             on        yes      /tmp
zroot/usr                             off       no       /usr
zroot/usr/home                        on        yes      /usr/home
zroot/usr/ports                       on        yes      /usr/ports
zroot/usr/src                         on        yes      /usr/src
zroot/var                             off       no       /var
zroot/var/audit                       on        yes      /var/audit
zroot/var/crash                       on        yes      /var/crash
zroot/var/log                         on        yes      /var/log
zroot/var/mail                        on        yes      /var/mail
zroot/var/tmp                         on        yes      /var/tmp

I am afraid that DATA datasets would be unmounted after the next reboot. As I still cannot find the cause of the previous accident.

SirDice · Friday at 8:28 AM

vienuolis said:
If DATA stands as a placeholder for MySQL and similar datasets, should not it be marked canmount=on, as ROOT is?

Look at zroot/var.

vienuolis said:
I am afraid that DATA datasets would be unmounted after the next reboot.

They're not. They will be mounted, the individual datasets (cloud, cyrus and mysql) are set to be mounted, it doesn't matter what the canmount property of the parent dataset says.

vienuolis · Friday at 9:11 AM

Yes, I see. However, canmount=off for /var implies its dependence on ROOT, which is actually canmount=on. Since I am scared, I am tempted to try DATA canmount=on, too. What could possibly go wrong with that?

vienuolis · Saturday at 3:06 PM

Perhaps the following explanation for this failure on reboot is possible:

zroot/ROOT canmount=on mountpoint=none;
zroot/DATA canmount=off mountpoint=none;
zroot/DATA/mysql canmount=on mountpoint=/var/db/mysql;
zroot/var/db — dataset does not exist;
zroot/var canmount=off mountpoint=/var;
zroot/ROOT canmount=on.

vermaden said:
So by default everything is included in BE - and if you want to EXCLUDE some parts of /usr or /var - you add needed ZFS datasets for these EXCLUDES.

SirDice said:
Yes, essentially zroot/var is a place holder, it's not mounted so it's actually part of the BE. That place holder just makes it easier to create zroot/var/mail for example and have it 'automagically' assume it should have /var/mail/ as its mountpoint.

SirDice said:
You need to have a zroot/var/spool or else you cannot create zroot/var/spool/cyrus. This is similar to files and directories, you cannot create /tmp/somedir/somefile if the /tmp/somedir directory doesn't exist.

Code:

> zfs list -o name,canmount,mounted,mountpoint
NAME                                  CANMOUNT  MOUNTED  MOUNTPOINT
zroot                                 on        yes      /zroot
zroot/DATA                            off       no       none
zroot/DATA/cloud                      on        yes      /usr/local/www/duo
zroot/DATA/cyrus                      on        yes      /var/spool/cyrus
zroot/DATA/mysql                      on        yes      /var/db/mysql
zroot/ROOT                            on        no       none
zroot/ROOT/14.3-R5p_2025-10-31good    noauto    no       /
zroot/ROOT/14.3-R5p_2025-10-30boot    noauto    yes      /
zroot/ROOT/14.3-R4p_2025-10-29        noauto    no       /
zroot/ROOT/14.3-R1p_2025-07-25        noauto    no       /
zroot/tmp                             on        yes      /tmp
zroot/usr                             off       no       /usr
zroot/usr/home                        on        yes      /usr/home
zroot/usr/ports                       on        yes      /usr/ports
zroot/usr/src                         on        yes      /usr/src
zroot/var                             off       no       /var
zroot/var/audit                       on        yes      /var/audit
zroot/var/crash                       on        yes      /var/crash
zroot/var/log                         on        yes      /var/log
zroot/var/mail                        on        yes      /var/mail
zroot/var/tmp                         on        yes      /var/tmp

Perhaps the parent /var/db needs to be mounted first in order for /var/db/mysql to be mounted. Or maybe it is enough to just set zroot/DATA canmount=on (as similar zroot/ROOT is set) avoiding the previous disaster on reboot.

Solved How to exclude some datasets from a boot environment

vienuolis

SirDice

Administrator

Erichans

vienuolis

Erichans

SirDice

Administrator

vermaden

vienuolis

SirDice

Administrator

vienuolis

vienuolis

vienuolis

vienuolis

vienuolis

vienuolis

vienuolis

vienuolis

vienuolis

SirDice

Administrator

vienuolis

vienuolis