ZFS ZFS snapshot - where are the files?

[sabotrax]

Hi,
I asked this first on Practical ZFS, but will repeat it here.

tldr; zfs snapshots of datasets are empty?

# uname -a
FreeBSD zabbix 14.1-RELEASE-p3 FreeBSD 14.1-RELEASE-p3 GENERIC amd64

# zfs list
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot                                          13.1G   216G    96K  /zroot
zroot/ROOT                                     13.1G   216G    96K  none
zroot/ROOT/default                             13.1G   216G  6.94G  /
zroot/tmp                                       148K   216G   148K  /tmp
zroot/usr                                       484K   216G    96K  /usr
zroot/usr/home                                  196K   216G   196K  /usr/home
zroot/usr/ports                                  96K   216G    96K  /usr/ports
zroot/usr/src                                    96K   216G    96K  /usr/src
zroot/var                                      3.02M   216G    96K  /var
zroot/var/audit                                  96K   216G    96K  /var/audit
zroot/var/crash                                  96K   216G    96K  /var/crash
zroot/var/log                                  2.43M   216G  1.90M  /var/log
zroot/var/mail                                  220K   216G   220K  /var/mail
zroot/var/tmp                                   104K   216G   104K  /var/tmp

# echo "lala" > /usr/afile.txt
# ls -l /usr/
total 255
-rw-r--r--   1 root wheel   5 Aug 28 19:18 afile.txt
drwxr-xr-x   2 root wheel 488 Aug  9 10:05 bin
drwxr-xr-x   4 root wheel   3 Aug 28 19:10 home
drwxr-xr-x  56 root wheel 356 Aug 13 13:23 include
drwxr-xr-x  11 root wheel 582 Aug 13 13:23 lib
drwxr-xr-x   9 root wheel 604 Aug 13 13:23 lib32
drwxr-xr-x   4 root wheel   4 Aug 13 13:23 libdata
drwxr-xr-x  12 root wheel  71 Aug  9 10:05 libexec
drwxr-xr-x  13 root wheel  15 Aug  9 14:17 local
drwxr-xr-x   2 root wheel   2 May 12  2022 obj
drwxr-xr-x   3 root wheel   2 Sep 15  2023 ports
drwxr-xr-x   2 root wheel 291 Aug  9 10:05 sbin
drwxr-xr-x  33 root wheel  33 Oct 31  2023 share
drwxr-xr-x   3 root wheel   2 May 12  2022 src
drwxr-xr-x  18 root wheel  18 Aug  9 09:44 tests

# zfs snapshot zroot/usr@now
# zfs list -t snapshot zroot/usr@now
NAME            USED  AVAIL  REFER  MOUNTPOINT
zroot/usr@now     0B      -    96K  -

# zfs get snapdir zroot/usr
NAME       PROPERTY  VALUE    SOURCE
zroot/usr  snapdir   visible  inherited from zroot
# ls -la /usr
total 289
drwxr-xr-x  16 root wheel  17 Aug 28 19:18 .
drwxr-xr-x  20 root wheel  25 Aug 14 14:14 ..
-rw-r--r--   1 root wheel   5 Aug 28 19:18 afile.txt
drwxr-xr-x   2 root wheel 488 Aug  9 10:05 bin
drwxr-xr-x   4 root wheel   3 Aug 28 19:10 home
drwxr-xr-x  56 root wheel 356 Aug 13 13:23 include
drwxr-xr-x  11 root wheel 582 Aug 13 13:23 lib
drwxr-xr-x   9 root wheel 604 Aug 13 13:23 lib32
drwxr-xr-x   4 root wheel   4 Aug 13 13:23 libdata
drwxr-xr-x  12 root wheel  71 Aug  9 10:05 libexec
drwxr-xr-x  13 root wheel  15 Aug  9 14:17 local
drwxr-xr-x   2 root wheel   2 May 12  2022 obj
drwxr-xr-x   3 root wheel   2 Sep 15  2023 ports
drwxr-xr-x   2 root wheel 291 Aug  9 10:05 sbin
drwxr-xr-x  33 root wheel  33 Oct 31  2023 share
drwxr-xr-x   3 root wheel   2 May 12  2022 src
drwxr-xr-x  18 root wheel  18 Aug  9 09:44 tests

Where is /usr/.zfs ?

Going on:

# zfs clone zroot/usr@now zroot/usr-clone
# zfs list zroot/usr-clone
NAME              USED  AVAIL  REFER  MOUNTPOINT
zroot/usr-clone     0B   216G    96K  /zroot/usr-clone
# ls -la /zroot/usr-clone/.zfs/snapshot/
total 0
dr-xr-xr-x+ 2 root wheel 2 Jan  1  1970 .
dr-xr-xr-x+ 3 root wheel 3 Sep 15  2023 ..

Where are the files?

I replicated the behavior on three FreeBSD machines.
If I do the same on zroot/var/tmp, which has no child sets, everything is ok.

Sanoid's Jim Salter hinted, that the child datasets might not be mounted correctly
because most of them only refer 96k.
But it's a default FreeBSD installation. I'm puzzled.

Thanks in advance
Marcus

 

[sabotrax]

I can hopefully answer my own question:
This has something to do with boot environments.

This is the active boot environment default:

# bectl list
BE                                 Active Mountpoint Space Created
.
..
default                            NR     /          13.2G 2023-09-15 14:44

Which is the dataset zroot/ROOT/default:

# zfs list -r zroot/ROOT
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot/ROOT                                     13.2G   215G    96K  none
.
..
zroot/ROOT/14.1-RELEASE-p3_2024-08-13_132308      8K   215G  5.01G  /
zroot/ROOT/default                             13.2G   215G  7.10G  /

Child datasets can be included in a boot environment or not.
Depending on the zfs property canmount.
zroot/usr is beeing included.

# zfs get canmount zroot/usr
NAME       PROPERTY  VALUE     SOURCE
zroot/usr  canmount  off       local

zroot/usr/home, for example, is not:

# zfs get canmount zroot/usr/home
NAME            PROPERTY  VALUE     SOURCE
zroot/usr/home  canmount  on        default

So, if you want to snapshot zroot/usr, you really have to snapshot zroot/ROOT/default.

I got fooled by the output of:

# zfs list zroot/usr
NAME        USED  AVAIL  REFER  MOUNTPOINT
zroot/usr   484K   215G    96K  /usr

Instead of looking at:

# mount|grep usr
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)

Greetings
Marcus

 

[SirDice]

zroot/usr is beeing included.

So, if you want to snapshot zroot/usr, you really have to snapshot zroot/ROOT/default.

No, /usr is included in zroot/ROOT/default, not zroot/usr, there is a difference here and you seem to equate the two. All datasets below zroot/usr (zroot/usr/src for example) aren't included in the BE. Same for zroot/var.