ZFS Organize ZFS for a desktop

Hello zfs on freebsd enthusiasts

I plan to (re)install FreeBSD 11 on a full ZFS pool, mostly for a desktop usage.
So, I wonder what ZFS sets tree would be more reliable.

On a server, I got that one:
  • root
  • /home
  • /tmp
  • /var
  • /var/cache
  • /var/crash
  • /var/db
  • /var/empty
  • /var/log
  • /var/mail
  • /var/run
  • /var/tmp

As it' s my intention to follow 11-Stable, I probably need the following:
  • /usr
  • /usr/ports
  • /usr/src
  • ... something to handle zfs boot environments ?

and, as I may create jails and/or bhyve guests:
/jail or /vm

The goal is to easily bundle data for separates backups and snaps but also to activate compression where it's suitable to.

How do you organize your zfs sets within your desktop ?
What are your advices and what point I have to consider or care about?
One of the beauty thing of ZFS is that you can add additional datasets later, as required.
You are not forced to do that on installation like when using traditional filesystems. ;)
One of the beauty thing of ZFS is that you can add additional datasets later, as required.

Trust me, things get pretty frustrating if you've got limited space, a running system, and find yourself needing to reorganize your datasets. Planning out ahead of time is definitely the way to go; I had to back up and restore several times before I had something I could stick with long-term on a single 250GB SSD.

Here's a breakdown of what I've got on my laptop. copies=2 and compress=lz4 are set on the pool at creation time, with only a few things explicitly excluded afterward (noted below).

The base system and applications are in one tree for use with boot environments, with /apps as a sub-tree so ports/packages can be included in boot environments, but rolled back without touching the base system. copies=1 is set on workbox/ROOT/master/src and workbox/ROOT/master/obj:
NAME                               MOUNTPOINT
workbox/ROOT                       none
workbox/ROOT/master                /
workbox/ROOT/master/apps           /usr/local
workbox/ROOT/master/apps/config    /usr/local/etc
workbox/ROOT/master/apps/pkgCache  /var/cache/pkg
workbox/ROOT/master/apps/pkgDB     /var/db/pkg
workbox/ROOT/master/obj            /usr/obj
workbox/ROOT/master/src            /usr/src
System stuff from /var that should be kept separate from boot environments:
workbox/VARIABLE        none
workbox/VARIABLE/crash  /var/crash
workbox/VARIABLE/log    /var/log
workbox/VARIABLE/mail   /var/mail
workbox/VARIABLE/tmp    /var/tmp
User data, with a dataset for all the top-level directories in /home/$USER:
NAME                    MOUNTPOINT
workbox/DATA            /usr/home/anoknusa
workbox/DATA/Design     /usr/home/anoknusa/Design
workbox/DATA/Downloads  /usr/home/anoknusa/Downloads
workbox/DATA/Edit       /usr/home/anoknusa/Edit
workbox/DATA/Organize   /usr/home/anoknusa/Organize
workbox/DATA/Write      /usr/home/anoknusa/Write
workbox/DATA/config     /usr/home/anoknusa/.config
workbox/DATA/mail       /usr/home/anoknusa/.local/var/mail
Media files. copies=1 is set here, because all data here is also is on my home server, multiple mobile devices, and the server is backed up weekly. This will change whenever I get around to buying a larger SSD:
NAME                      MOUNTPOINT
workbox/MEDIA             none
workbox/MEDIA/Comics      /usr/home/anoknusa/Read/Comics
workbox/MEDIA/Ebooks      /usr/home/anoknusa/Read/Ebooks
workbox/MEDIA/Music       /usr/home/anoknusa/Listen/Music
workbox/MEDIA/SpokenWord  /usr/home/anoknusa/Listen/SpokenWord
workbox/MEDIA/StandUp     /usr/home/anoknusa/Listen/StandUp

Weekly snapshots are made of workbox/ROOT/master. Daily snapshots are also made of workbox/ROOT/master/apps. Weekly and daily snapshots are made of workbox/DATA, and hourly snapshots are also made of workbox/DATA/Edit, workbox/DATA/Organize, workbox/DATA/Write, and workbox/DATA/mail. No snapshots are made on workbox/MEDIA, so as to exclude it from backups (I would end up with duplicates of each library on my home server, for no good reason). When I'm home, everything is automatically backed up to my home server using sysutils/zxfer every hour.
Trust me, things get pretty frustrating if you've got limited space

I didn't mean to suggest that no planning is required, but certainly there are less strict requirements when using ZFS.

Things would have been however frustrating when running on limited free storage, and much more when using conventional filesystem.

Thanks for sharing your hints about boot environments and backup management.
You can reorganize a Root-on-ZFS pool without a reinstall if you wish but you have to boot from a recovery media such as the FreeBSD memstick installation image in order to be able to relocate the root filesystem. I've done such reorganization a few times and it's definitely not too difficult if you just have a clear mind of what is going to moved and where.
Things would have been however frustrating when running on limited free storage, and much more when using conventional filesystem.

I'm not sure what it is you think is more frustrating; I'm guessing you're thinking in terms of partitions. What I'm referring to is something like this: say I have a ZFS filesystem. There is a directory sitting on that filesystem, subject to that filesystem's properties. Perhaps I don't want this directory subjected to those properties, though---say, I have copies=2 set, but this directory holds a large music collection or a bunch of virtual machine images that will eat up a lot of disk space. I don't want (and can live without) redundant copies of all that on my desktop/laptop, so I resolve to move them to a dedicated filesystem. Creating the filesystem can be simple enough, but then I have to move the files into that new filesystem. I'd have to mount the filesystem elsewhere, and either copy or move the files to it. I could use something like rsync to cleanly copy the files, but would temporarily have three copies of each file on the filesystem---hardly a solution to my original problem. I could also just mv the directory, or use rsync and immediately delete the original directory if I have enough space to do so. But once I've moved or deleted the original directory it would remain in the most recent snapshot of its original parent filesystem. I could get rid of it (most of it, anyway) by destroying the snapshot, but that would also entail destroying the recent backups of any other modified/deleted files that also reside on that filesystem. We might suppose that, for the time being, I have enough space to work with that I can just leave the snapshot be and wait for my automated snapshot schedule to rotate it out. But the number of snapshots containing data that refers to those files is a varaible that depends on both the snapshot schedule I've configured, and the span of time that directory has been subject to that snapshot schedule: I could have a half-dozen snapshots containing data from that directory, and they may run on a schedule that includes monthly or multiple weekly snapshots. So completely removing the original data once it's been copied to the new filesystem would entail either destroying many snapshots, or waiting an indefinite amount of time (possibly a month or more) for the old snapshots referring to the directory to be rotated out. In the meantime, that space will remain consumed. And that's about as simple as such a scenario gets. It can potentially get more complicated depending on how your ZFS dataset tree and filesystem organization interact, how frequently data is modified, what sort of backup scheme you want, etc.

Obviously, these things really only apply if you're working with a lot of data on a single disk that's, say, less than 1TB in size. If you're using a 1TB mirror you'll have much less to worry about, but that's often not the case on a desktop or laptop. It also drives home the point that while ZFS can certainly be used anywhere, it was really designed with high-capacity storage servers in mind. On the plus side, if you fall into the same traps I did you'll learn a hell of a lot about both the power and limitations of ZFS. :p

Well, that's well known but what are the consequences when upgrading/rollback systems when there are own datasets for /usr, /etc, /var?

They get excluded. Go back and look at my example: workbox/ROOT/master is the root of the boot environment tree. Only filesystems under that are included in the boot environment.
I just use separate / and /home with lz4 compression on top with beadm compatible dataset paths.
Complex layout structure is mostly UNIX history.
I'm not sure what it is you think is more frustrating; I'm guessing you're thinking in terms of partitions. What I'm referring to is something like this: [... etc...]

Yes, I was thinking in term of partitions / datasets.

I found interesting your layout/setup, but personally I would refrain to perform any complex setup, I've learned long ago to think at backup in terms of "time to restore" the system, also I like to keep the things as simple as possible. That's said, I'm still learning about ZFS, and I will certainly grab something from your example. :D
Can you please explain why this should be kept separate when switching boot environments?

They're directories containing files that shouldn't be strictly bound to one boot environment or another. While those things would be preserved in the boot environment, the state of those directories can affect the state of a running system or how you manage your work. A failed base system update will affect the base system, but /var/log contains logs for ports you've installed as well, and logs often contain information that's chronologically sensitive: you don't want today's security log destroyed when you revert to yesterday's system state. /var/audit is only used if you've configured system auditing, but again, that's information you always want a complete history of. You don't want to roll back your system mail spool, either. And it's just more convenient to keep all your kernel crash dumps and the stuff stored in /var/tmp in one place, rather than scattered across boot environments you then have to mount and hunt through to find what you want. And so on.

The installer includes /usr/local and other ports-related directories in boot environments, but that's a matter of taste, really. It only ever makes a difference when you track -CURRENT, or when the time comes for a major version upgrade.