ZFS snapshot directory access Operation not permitted in Jail on FreeBSD13

M

Mike Selner

Hi,

I recently upgraded some systems from FreeBSD12 to FreeBSD13 (OpenZFS).

Inside a jail, I get "Operation not permitted" when trying to access a directory. All access as root.

Code: 
ls /.zfs/snapshot/xxxx

On 11 & 12, I can view the contents of this directory inside the Jail. On 13, Operation not Permitted.

Outside the jail (host system), I can view the contents of this directory on all versions.

My jails are started/stopped by ezjail. Snapshots are done manually, not using ezjail.

I do not think it is related to the ZFS property "snapdir" which has always been "hidden" on all my datasets.
I believe snapdir: visible only affects the display of .zfs in a directory listing. I've always been able to explicitly access /.zfs/snapshot/xxxxx in the jail prior to 13.

I don't need to manage the dataset within the jail so I've not used the "jailed" property in the past, and it is "off" on all datasets.

Has anyone experienced this and is there a known fix?

Thanks
 
If I'm understanding correctly, the snapshot directory is on the host, accessing from the host works, but inside the jail you can't?
Don't jails effectively have a different "/" than the host? (chrooted as a way of describing it)

If the above describes it correctly, I'm not sure it's technically a bug (If I'm wrong, I'll defer to those with the knowledge).
 
Yes - the jail is chrooted. The .zfs directory is under the root inside the jail: /.zfs/snapshot/xxx
It was always available prior to 13. I suspect a change in the ZFS implementation.
 
I have found the same problem, but after accessing the snapshot directory from the host, the jail can access it.
 
After execute ls -li command in host for each .zfs/snapshot folder everything is back to normal.
Now I can execute ls -li command inside the jail for .zfs/snapshot without permission problem.

??


It's also resolve problem with shadow_copy2 in Samba. In my case, the previous versions stopped working after upgrade to FreeBSD 13.
 
I can confirm that compiling kernel13.2 you can access snapshot directory from inside jail without problem(with 13.1 fails), but you cant umount or destroy snapshots, i dont know if it can be related or it is another bug?
 
My original problem above still exists on 13.2-RELEASE-p2. kr0m may have a different issue resolved.

I cannot ls -al /.zfs/snapshot/* from within the jail until I do this on the host:

ls -al /var/jail/*/.zfs/snapshot/*
 
These are the steps that I took to get it working, maybe it is useful for you.
PARENT HOST:
uname -a
FreeBSD MightyMax.alfaexploit.com 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 releng/13.2-n254627-4341433a673f KR0M-MINIMAL amd64
Click to expand...
NOTE: I use a custom kernel, but nothing tweaked about ZFS.
zfs create -o mountpoint=/var/db/postgres/data15 -o canmount=noauto zroot/data_PostgreSQL00
zfs set jailed=on zroot/data_PostgreSQL00

bastille config PostgreSQL00 set allow.mount=1
bastille config PostgreSQL00 set allow.mount.zfs=1
bastille config PostgreSQL00 set enforce_statfs=1

zfs jail PostgreSQL00 zroot/data_PostgreSQL00
Click to expand...
INSIDE JAIL:
sysrc zfs_enable=YES
zfs mount zroot/data_PostgreSQL00

zfs snapshot zroot/data_PostgreSQL00@backup
zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zroot/data_PostgreSQL00@backup 0B - 12.3M -

ls -la /var/db/postgres/data15/.zfs/snapshot/
total 9
dr-xr-xr-x+ 3 root wheel 3 Mar 5 08:22 .
dr-xr-xr-x+ 3 root wheel 3 Mar 5 08:11 ..
drwx------ 19 postgres postgres 26 Mar 5 08:16 backup

umount /var/db/postgres/data15/.zfs/snapshot/backup
zfs destroy zroot/data_PostgreSQL00@backup
Click to expand...
 
I am sorry to inform that I made some mistake or something, because now I am getting the same error before.
I can take snapshots, access It's content but I can't umount snapshots dir nor delete snapshots:
umount /var/db/postgres/data15/.zfs/snapshot/backup
umount: unmount of /var/db/postgres/data15/.zfs/snapshot/backup failed: Operation not permitted
Click to expand...
zfs destroy zroot/data_PostgreSQL00-test@backup
cannot destroy snapshot zroot/data_PostgreSQL00-test@backup: dataset is busy
Click to expand...
 
Something weird is happening, after a fresh reboot of physical server, I start my jail, dataset is not mounted automatically, I mount it manually. Snapshots creation, umount and snapshot deletion works.
If I restart the jail, dataset is mounted automatically, snapshots creation continues working but umount and snapshot deletion doesn't works anymore.
 
I have found the problem.
The mount problem was my own mistake, not related to ZFS dataset delegation to jails.

In the other hand the key is to reset "zfs set jailed" property every time the jail is started.
PostgreSQL01-test {
devfs_ruleset = 4;
enforce_statfs = 1;
exec.clean;
exec.consolelog = /var/log/bastille/PostgreSQL01-test_console.log;
exec.created = 'zfs set jailed=off zroot/data_PostgreSQL01-test && zfs set jailed=on zroot/data_PostgreSQL01-test && zfs jail PostgreSQL01-test zroot/data_PostgreSQL01-test';
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = PostgreSQL01-test;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/PostgreSQL01-test/fstab;
path = /usr/local/bastille/jails/PostgreSQL01-test/root;
securelevel = 2;

interface = nfe0;
ip4.addr = 192.168.69.27/24;
ip6 = disable;
sysvmsg=new;
sysvsem=new;
sysvshm=new;
allow.mount=1;
allow.mount.zfs=1;
}
Click to expand...

In resume, the jail parameters needed to get it working are:
  • enforce_statfs = 1;
  • exec.created = 'zfs set jailed=off zroot/data_PostgreSQL01-test && zfs set jailed=on zroot/data_PostgreSQL01-test && zfs jail PostgreSQL01-test zroot/data_PostgreSQL01-test';
  • allow.mount=1;
  • allow.mount.zfs=1;
The dataset creation command:
  • zfs create -o mountpoint=/var/db/postgres/data15 zroot/data_PostgreSQL01-test
An inside jail commands:
  • sysrc zfs_enable=YES
Here I paste my own example jail:
PostgreSQL01-test # ~> zfs list -t snapshot
no datasets available

PostgreSQL01-test # ~> zfs snapshot zroot/data_PostgreSQL01-test@backup

PostgreSQL01-test # ~> ls -la /var/db/postgres/data15/.zfs/snapshot/backup
total 94
drwx------ 19 postgres postgres 29 Sep 4 10:05 .
dr-xr-xr-x+ 3 root wheel 3 Sep 4 10:05 ..
-rw------- 1 postgres postgres 3 Sep 3 14:07 PG_VERSION
-rw------- 1 postgres postgres 226 Sep 3 14:07 backup_label.old
-rw------- 1 postgres postgres 180969 Sep 3 14:07 backup_manifest
drwx------ 7 postgres postgres 7 Sep 3 17:00 base
drwx------ 2 postgres postgres 62 Sep 3 17:47 global
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_commit_ts
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_dynshmem
-rw------- 1 postgres postgres 4931 Sep 3 14:07 pg_hba.conf
-rw------- 1 postgres postgres 1636 Sep 3 14:07 pg_ident.conf
drwx------ 4 postgres postgres 5 Sep 4 09:49 pg_logical
drwx------ 4 postgres postgres 4 Sep 3 14:07 pg_multixact
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_notify
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_replslot
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_serial
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_snapshots
drwx------ 2 postgres postgres 2 Sep 4 10:05 pg_stat
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_stat_tmp
drwx------ 2 postgres postgres 3 Sep 3 14:12 pg_subtrans
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_tblspc
drwx------ 2 postgres postgres 2 Sep 3 14:07 pg_twophase
drwx------ 3 postgres postgres 5 Sep 3 14:12 pg_wal
drwx------ 2 postgres postgres 3 Sep 3 14:07 pg_xact
-rw------- 1 postgres postgres 339 Sep 3 14:07 postgresql.auto.conf
-rw------- 1 postgres postgres 29466 Sep 3 14:07 postgresql.conf
-rw------- 1 postgres postgres 55 Sep 4 10:05 postmaster.opts
-rw------- 1 postgres postgres 90 Sep 4 10:05 postmaster.pid
-rw------- 1 postgres postgres 0 Sep 3 14:07 standby.signal

PostgreSQL01-test # ~> mount -v |grep backup
zroot/data_PostgreSQL01-test@backup on /var/db/postgres/data15/.zfs/snapshot/backup (zfs, local, noatime, nosuid, read-only, nfsv4acls)

PostgreSQL01-test # ~> zfs destroy zroot/data_PostgreSQL01-test@backup

PostgreSQL01-test # ~> zfs list -t snapshot
no datasets available
Click to expand...
 
You must log in or register to reply here.
Back
Top