ZFS ZFS root from sysinstall ... why so many filesystems ?

[absduser]

I just created a new system from sysinstall (used the 12.0-RELEASE ISO) and chose to use the nice, menu driven ZFS boot config that is now in sysinstall.

Everything went fine and I am booted up, etc. however I am really surprised to see all of the individual filesystems that were created for me ...

I really don't think I need different filesystems for basically every directory inside of /var ... and for /src ?

What is the thinking behind all of this and is it possible for me to configure a ZFS boot config that just has a /usr and /var as separate filesystems since that is all that I need ?

Thanks.

 

[k.jacker]

I guess the datasets provided by the installer are mainly ment as a good starting point for easy creation of ZFS snapshots.
Keep in mind they are ment to suite any FreeBSD user, not just you.
Change it using zfs(8) if you don‘t like it. Keep in mind that on a root-on-zfs some datasets are not replacable by a plain directory while FreeBSD is running.
Boot config has nothing to do with that.

 

[olli@]

Also keep in mind that ZFS filesystems are rather cheap (compared to traditional filesystems like UFS) and take near zero resources. You can create 1000 of them if you want.

 

[ralphbsz]

I really don't think I need different filesystems for basically every directory inside of /var ... and for /src ?

What is the thinking behind all of this and is it possible for me to configure a ZFS boot config that just has a /usr and /var as separate filesystems since that is all that I need ?

Need? You really don't need separate file systems at all. They are there for the following reasons though. A Unix system requires certain directories to exist and to be writable in order to function reasonably well. That includes at least /tmp, various parts of /var, and in most cases also /etc plus the home directory of the users (which are typically in /usr/home or /home). The home directory of the root user (which is, for good reason, in the root directory) is particularly important, because if root can't log in, root can't repair things. If those areas become unwritable due to being overfilled, the system or its users will be unhappy.

Having separate file systems means that if a rogue process or user overfills a file system, or attempts to do that, it is easier to diagnose what is going wrong, cure it, and less likely that all of the system goes down. It also means that normal users (not logged in as root) are basically incapable of performing the denial-of-service attack of overfilling shared file systems.

Underlying all this is the assumption that the system is mostly there to be used and to serve the users (not just as a toy to be administered and played with), which implies that reliability, and ease of administration is of foremost importance.

If all users (including the administrator) of the system were hyper-intelligent and careful, this would make no difference, and a single file system would be sufficient, and more efficient and simpler. Similarly, if the owner of the system does't care about uptime, reliability, or resilience (and reinstalling the OS is considered acceptable), then multiple file systems don't do anything useful.

What this points out is that the *BSD family of systems comes from a certain heritage and culture, namely shared multi-user server systems, whose main purpose is to dependably store data. This is quite different from other systems, which were born and grown in the tradition of single-user personal machines, with valuable data stored (at least redundantly) off-system.

if you don't like the complex file system layout, you can fix that on the next install, based on your use case.

 

[driesm]

Don't forget that the pooled behavior of ZFS is probably its biggest benefit, I suggest that you read up on ZFS in general.
The classic "filesystem" like UFS where size is fixed by the partition you create it on doesn't necessarily apply to ZFS.
ZFS just has a big pool of data that can be shared by different "mini filesystems" so called "datasets".

Each dataset has the ability to have different properties which is why you typically want as many as possible!
For example: you can set a higher compression level on the zroot/var/log dataset because that dataset will contain text files which are highly compressible. You can increase the blocksize of a dataset to 1MB becasue it will store primarily large files such as videos, photos to reduce metadata usage. All of the above is not possible when you only have one "filesystem" or "dataset".

Also, the reason why /var and /usr are created is only for inheriting properties so that child datasets get a correct mountpoint, eg when you create zroot/usr/src, it will be mounted under /usr/src. As you will be able to see with the following commands they are actually not even mounted:

zfs get canmount zroot/var
zfs get canmount zroot/usr

So you might wonder, where the hell are my folders and files that reside under /usr stored then? They're definitely not under zroot/usr, as its not mounted. They actually reside on your current boot environment, where the default is root/ROOT/default, take a look at bectl().

There has really gone A LOT of thought in the default dataset layout. I would even state that the installer creates a minimal layout.
There is no downside to having more datasets, there are only benefits and a higher level of control.
It might seem complex at first but it makes sense once you dive into ZFS.

I have about 20-ish datasets living under zroot ATM.

 

[SirDice]

I have about 20-ish datasets living under zroot ATM.

I can top that, and I'm not even doing anything 'special':

dice@molly:~ % zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
fbsd1                              1.01T   766G   117K  none
fbsd1/DATA                          917G   766G   117K  none
fbsd1/DATA/MAME                     850G   766G   850G  /storage/MAME
fbsd1/DATA/commodore               66.7G   766G  66.7G  /storage/commodore
fbsd1/DATA/mysql                    704M   766G   704M  /var/db/mysql
fbsd1/jails                        4.28G   766G   117K  /jails
fbsd1/jails/clean_jail             4.24G   766G  2.27G  /jails/clean_jail
fbsd1/jails/j-ports                47.2M   766G  2.31G  /jails/j-ports
fbsd1/obj                          20.9G   766G  20.9G  /usr/obj
fbsd1/ports                        41.0G   766G  3.48G  /usr/ports
fbsd1/ports/distfiles              35.8G   766G  35.8G  /usr/ports/distfiles
fbsd1/ports/packages               12.2M   766G  12.2M  /usr/ports/packages
fbsd1/poudriere                    28.8G   766G   117K  none
fbsd1/poudriere/data               20.3G   766G  20.3G  /usr/local/poudriere/data
fbsd1/poudriere/jails              4.14G   766G   117K  none
fbsd1/poudriere/jails/113-release  1.17G   766G  1.17G  /usr/local/poudriere/jails/113-release
fbsd1/poudriere/jails/12-stable    1.41G   766G  1.41G  /usr/local/poudriere/jails/12-stable
fbsd1/poudriere/jails/120-release  1.56G   766G  1.56G  /usr/local/poudriere/jails/120-release
fbsd1/poudriere/ports              4.33G   766G   117K  none
fbsd1/poudriere/ports/desktop      1.88G   766G  1.88G  /usr/local/poudriere/ports/desktop
fbsd1/poudriere/ports/server       2.46G   766G  2.46G  /usr/local/poudriere/ports/server
fbsd1/release                      6.64G   766G  3.62G  /storage/release
fbsd1/release/12-stable            3.02G   766G  3.02G  /storage/release/12-stable
fbsd1/src                          7.84G   766G  5.57G  /usr/src
fbsd1/vm                           3.64G   766G  2.93G  /vm
fbsd1/vm/images                     723M   766G   723M  /vm/images
storage                            7.66T  3.10G   209K  /storage
storage/DayZ                        163K  3.10G   163K  /storage/DayZ
storage/backups                    21.6G  3.10G  21.5G  /storage/backups
storage/backups/bareos             52.4M  3.10G  52.4M  /storage/backups/bareos
storage/media                      7.63T  3.10G  7.63T  /storage/media
storage/test                        209K  3.10G   209K  /storage/test
zroot                              85.6G   348G    31K  none
zroot/HOMES                        40.7G   348G  26.5G  /usr/home
zroot/ROOT                         44.8G   348G    31K  none
zroot/ROOT/12-stable               44.8G   348G  28.7G  /

This is a really old install though, I really should re-install some time. I think I set this up with 8.0, by hand (installer didn't have the ZFS option back then).

 

[absduser]

Thanks for these replies and examples.

Let's say I wanted fewer than the sysinstall-provided filesystems ... maybe I don't think I need so many different /var/xxx filesystems or whatever ... is there a way to direct that *from within sysinstall* and still be able to use the nice zfs-root sysinstall options ?

Or do I have to go fully hand made filesystem at that point ?

Thanks.

 

[`Orum]

is there a way to direct that *from within sysinstall* and still be able to use the nice zfs-root sysinstall options ?

Or do I have to go fully hand made filesystem at that point ?

This depends on what you mean by "within sysinstall" (and by sysinstall I assume you mean bsdinstall(8)). If you want a nice GUI that walks you through it, no. However, you can simply exit to a shell, do whatever your heart desires, and then resume with bsdinstall from that point on. I do this often as there are many times I don't want ZFS on root to take up all the space on my disks, so it must be done manually.

However, I really question why you would do this to remove the default datasets. Many of them are extremely useful to avoid backing up data you don't need to back up (e.g. /usr/ports) or to change ZFS options on a per-dataset basis (also useful for /usr/ports among others). Be absolutely sure you know what you are doing as this can impact security as well, for example the setuid=off in several places.

 

[absduser]

However, I really question why you would do this to remove the default datasets. Many of them are extremely useful to avoid backing up data you don't need to back up (e.g. /usr/ports) or to change ZFS options on a per-dataset basis (also useful for /usr/ports among others). Be absolutely sure you know what you are doing as this can impact security as well, for example the setuid=off in several places.

I totally agree with you ... to a point ... but that list is really long - do I really need 15 datasets ? Do I really need a dataset for basically every single directory in /var ? My use-case for a lot of servers is very, very minimal and I don't really need to see a page of output every time I list mount points or 'zfs list' and so on ...

 

[D-FENS]

I just created a new system from sysinstall (used the 12.0-RELEASE ISO) and chose to use the nice, menu driven ZFS boot config that is now in sysinstall.

Everything went fine and I am booted up, etc. however I am really surprised to see all of the individual filesystems that were created for me ...

I really don't think I need different filesystems for basically every directory inside of /var ... and for /src ?

What is the thinking behind all of this and is it possible for me to configure a ZFS boot config that just has a /usr and /var as separate filesystems since that is all that I need ?

Thanks.

What you are seeing on a clean installation is not many by any measure.
Take a look what I have on a very small server I am supporting:

root@???~ # zfs list | wc -l
     460

It has 460 datasets and this is by no means an extreme value. With a slightly larger server I could easily have 5000 datasets.

You should not compare ZFS datasets to conventional disk partitions. Datasets are quite cheap and you can easily make thousands of them. They offer the possibility to make snapshots, rollback, send and receive the whole dataset to another system or back up to a file. You could set different properties for your different datasets - for example is it compressed, with which algorithm, is it deduplicated, etc. You can set different permissions for different datasets. You can clone datasets into new ones almost for free.
It's a completely new paradigm and by putting everything in one big dataset you lose 90% of the features that ZFS offers you.

It's probably best to pick up a ZFS book or a tutorial and you'll see the advantages.

 

[D-FENS]

One more thing. To explain the datasets of a new FreeBSD installation - look into Boot Environments. Basically, you could have more than one FreeBSD system installed in the same ZFS pool. You could install for example 12.0-RELEASE, 12.1-RELEASE and 11.3-RELASE in parallel. And you can switch between them on the fly with beadm(1). This way you can test new versions or updates without risking your production environment. Just clone the environment into a new one, update it and test. If something goes wrong, destroy the clone and you are back online.

 

[ralphbsz]

I totally agree with you ... to a point ... but that list is really long - do I really need 15 datasets ? Do I really need a dataset for basically every single directory in /var ? My use-case for a lot of servers is very, very minimal and I don't really need to see a page of output every time I list mount points or 'zfs list' and so on ...

Need? No. You can survive without. Matter-of-fact, if you want to use ZFS exactly like people have used traditional file systems, the data sets don't give you anything.

But they also don't cost anything. Other than upsetting you when you have to look at a screenful of them. I would suggest that you learn to live with that, but if you want, you can also change your configuration manually.

But you have to remember that data sets are not really like mount points. Yes, they are places where the file system structure and options change, like mount points in a traditional file system. But they are not the point where space allocation changes (one data set does not run out of space like a file system does), nor do they correspond to underlying storage devices. That aspect of traditional file systems is the zpool. Perhaps the correct lesson might be this: instead of doing "zfs list" as a replacement for "mount" and "df", you should get into the habit of doing "zpool list". This might help minimize the annoyance of having to look at a long list, and also perhaps give you more of the information you really want at the fingertips.

 

[`Orum]

I totally agree with you ... to a point ... but that list is really long - do I really need 15 datasets?

If you think 15 is a lot you should see some of the servers I administrate...

Anyway, if you don't need them, simply don't worry about them. If you really, REALLY want to get rid of them, you can delete them after installation assuming they're empty, or move the files around if they're not. Alternatively drop into a shell and do all the zpool/zfs configuration you want by hand. Honestly, there many, many more egregious things in FreeBSD by default than a few extra datasets that are beyond negligible on my radar.

 

[D-FENS]

the data sets don't give you anything.

That's not completely true. There are quite a few benefits, I described them here and here

That aspect of traditional file systems is the zpool. Perhaps the correct lesson might be this: instead of doing "zfs list" as a replacement for "mount" and "df", you should get into the habit of doing "zpool list". This might help minimize the annoyance of having to look at a long list, and also perhaps give you more of the information you really want at the fingertips.

Good point! I would add also zfs mount to the list.
Also, quite useful is using zfs list -d1 for showing only the top level (works with snapshots, as well as datasets) and zfs list -o space for showing used space.

 

[absduser]

Thank you.

All of these points are very well taken and I will most likely just keep the default dataset layout.

I also have some additional datasets for my own purposes that I will probably add.

However, just because I am curious, if I *did want to rip out a dataset* ... for instance, let's say I wanted /var/empty to just be a plain old directory under /var ...

Something like:

zfs destroy zpool/var/empty
mkdir /var/empty

... is that it ?

 

[D-FENS]

Yes, you could do that.
If the directory is not empty, you have to copy it first to the dataset, obviously. You could do that by copying to /var/nonempty2, then destroy the dataset and rename to /var/nonempty.
Or you could change the mountpoint of the dataset to /tmp/nonempty and then copy to /var/nonempty.

One thing to keep in mind - if you convert a dataset to a directory in another dataset, you lose that dataset's history (snapshots). So you cannot go back and compare versions of files in the past.