Understanding permissions in jails and host

Hello everyone,

I have recently upgraded my FreeBSD 10.3 server to 11.2.
With this change, I also setup my system using jails:

- Plex
- Samba
- Apache PHP stack
- Nginx Node.js stack
- Database stack
- Etc.

I have two ZFS pools:
- zfs-os the FreeBSD OS
- data mounted to /data

I am using sysutils/iocage as tool for managing my jails.

The issue

After setting up my jails, I mounted bits of the data pool to some of my jails using nullfs:
iocage fstab -a plex "/data/media /plexdata nullfs rw 0 0"

After mounting, I restarted and went into the jail and set the rights to a local user, for plex this was the user plex:
chown -R plex:plex /plexdata

I then continued to setup the other jails, ending with the samba jail:
iocage fstab -a samba "/data /share nullfs rw 0 0"
chown -R user:group /share

I then started using the system and everything seemed fine, the samba was working and other jails were working fine too.
But, when I later wanted to use the plex server, the plexdata folder and its subfolders were inaccessible.
I noticed that the plexdata directory had different user/group permissions, set to a GUID:
drwxrwxr-- 6 972 972 6 Nov 13 2016 plexdata

After changing the permissions of the folder recursive back to user plex, group plex, samba could not access the directory anymore?
Also, I noticed that the /data directory on my host had changed its permissions to the GUID of the plex user.

I thought that each jail has it's own permission system and could not change things on the host.
I also thought that mounting a directory to a jail using nullfs would create a different access point, without collision of permissions.
Apparently I was wrong.

Is there a way to get this working?
And could you please explain me why using jails and mounting a directory still changes permissions on host/other jail?
First you need to realize that a Jail is not some kind of virtual machine which is fully shielded from the host, it's merely an extra kernel subprocess which runs on top of another userland. And while the Jail uses its own user database it's still running on top of your main host and sharing the same file system(s).

Another important aspect is nullfs mounts. If you apply permissions to a file system then those permissions also apply to that filesystem, not the underlying mountpoint. Also: if you mount a filesystem then any permission settings on the mountpoint itself will be overruled by the filesystem:
root@zefiris:/home/peter # mkdir mysource && chmod 777 mysource
root@zefiris:/home/peter # ls -ld mysource
drwxrwxrwx  2 root  peter  2 Oct  6 15:08 mysource/
root@zefiris:/home/peter # mount -t nullfs /usr/src mysource/
root@zefiris:/home/peter # ls -ld mysource/
drwxr-xr-x  25 root  wheel  38 Sep  6 10:56 mysource//
So if I were to change the permissions on mysource I'm not changing the mountpoint but the underlying file system. Which in this case is actually /usr/src. Mounting something using nullfs is not the same as making a symlink, I think you might be confusing those aspects.

Therefor it you were to mount /usr/src using nullfs on multiple locations then they'll all share the same filesystem and therefor the same permission settings. This is basically the same as exporting a filesystem using NFS for a remote host while the filesystem uses UID/GID's above 1000 (so custom user & groups). While you may see user 'peter' as the owner a remote host would probably see 1020 or maybe even an entirely different name.

This also applies to Jails. Although a Unix environment (such as a jail) may use a separate user database it'll still be using the same mechanics for handling those permissions: UID's and GID's. The only difference is that the associated account names might be different.

And that's basically all there is to your problem, hope this shed some light on it.


Fixing the problem... you could start by setting up the same user accounts across all hosts. Then you won't risk overlap. And of course: use different filesystems, don't create cross-overs if you don't have to. I don't use Jails too often but when I do I set up a dedicated filesystem for it to create an extra "layer" of separation.