Other Files and folders not visible when mounted will nullfs

I am running a FreeBSD 11.1 (ZFS) system with jails and I mount a file system (/storage/cloud) on my storage pool into the jail (/jails/cloud/storage) via nullfs(4), and that works fine for what the jail does. However I just noticed that outside the jail, I can only see the mount points of the file system children, but not the contents.

For example if I do: ls -la /storage/cloud/* (As root) then nothing shows up, but they are present inside the jail. Even if I stop the jail, they still are not present outside the jail.

I just tried disabling jails in /etc/rc.conf and rebooting the system and EVEN THEN the files do not reappear outside the jail in their original location, yet zfs list reports the space being used. But if I then (without rebooting) enable the jail, the files show up in the nullfs-mounted location inside the jail!

As an experiment, I tried creating /mnt/test and /mnt/test2 and creates a folder and a few files in test, and then nullfs-mounted it onto test2, and I could then see the files in both locations.

Does anybody know what might be causing this?
 
My guess would be that your initial nullfs(4) failed and the files are actually stored inside the jail's directory and not on the nullfs(4) mounted filesystem.
 
My guess would be that your initial nullfs(4) failed and the files are actually stored inside the jail's directory and not on the nullfs(4) mounted filesystem.
I just double-checked, and that is not the case, because the storage usage reported by zfs list shows that the data is stored on the storage pool:

Code:
# zfs list storage
NAME      USED  AVAIL  REFER  MOUNTPOINT
storage  56.0G  20.9T   176K  /storage
# zfs list system/jails
NAME           USED  AVAIL  REFER  MOUNTPOINT
system/jails  1.92G  79.5G    88K  /jails

The weird/worrying thing also is that this problem survives a reboot (With jails disabled).
 
Can you post the output of mount (on the host) when the jail is stopped and when the jail is running?
 
Can you post the output of mount (on the host) when the jail is stopped and when the jail is running?
Yep, you'll find it below along with a diff for convenience.

Diff:
Code:
36a37,38
> /storage/cloud on /jails/cloud/storage (nullfs, local)
> devfs on /jails/cloud/dev (devfs, local, multilabel)


Jail stopped:
Code:
system/root on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
system/root/home on /home (zfs, local, noatime, nfsv4acls)
system/root/tmp on /tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr on /usr (zfs, local, noatime, nfsv4acls)
system/root/usr/local on /usr/local (zfs, local, noatime, nfsv4acls)
system/root/usr/obj on /usr/obj (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr/ports on /usr/ports (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr/src on /usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var on /var (zfs, local, noatime, nfsv4acls)
system/root/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/mail on /var/mail (zfs, local, noexec, nosuid, nfsv4acls)
system/root/var/tmp on /var/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails on /jails (zfs, local, noatime, nfsv4acls)
system/jails/cloud on /jails/cloud (zfs, local, noatime, nfsv4acls)
system/jails/cloud/home on /jails/cloud/home (zfs, local, noatime, nfsv4acls)
system/jails/cloud/tmp on /jails/cloud/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr on /jails/cloud/usr (zfs, local, noatime, nfsv4acls)
system/jails/cloud/usr/local on /jails/cloud/usr/local (zfs, local, noatime, nfsv4acls)
system/jails/cloud/usr/obj on /jails/cloud/usr/obj (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr/ports on /jails/cloud/usr/ports (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr/src on /jails/cloud/usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var on /jails/cloud/var (zfs, local, noatime, nfsv4acls)
system/jails/cloud/var/audit on /jails/cloud/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/crash on /jails/cloud/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/log on /jails/cloud/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/mail on /jails/cloud/var/mail (zfs, local, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/mysql on /jails/cloud/var/mysql (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/tmp on /jails/cloud/var/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/www on /jails/cloud/var/www (zfs, local, noatime, noexec, nosuid, nfsv4acls)
storage on /storage (zfs, local, noatime, nfsv4acls)
storage/cloud on /storage/cloud (zfs, local, noatime, nfsv4acls)
storage/cloud/bc on /storage/cloud/bc (zfs, local, noatime, nfsv4acls)
storage/cloud/jocmei on /storage/cloud/jocmei (zfs, local, noatime, nfsv4acls)

Jail running:
Code:
system/root on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
system/root/home on /home (zfs, local, noatime, nfsv4acls)
system/root/tmp on /tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr on /usr (zfs, local, noatime, nfsv4acls)
system/root/usr/local on /usr/local (zfs, local, noatime, nfsv4acls)
system/root/usr/obj on /usr/obj (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr/ports on /usr/ports (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/usr/src on /usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var on /var (zfs, local, noatime, nfsv4acls)
system/root/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/root/var/mail on /var/mail (zfs, local, noexec, nosuid, nfsv4acls)
system/root/var/tmp on /var/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails on /jails (zfs, local, noatime, nfsv4acls)
system/jails/cloud on /jails/cloud (zfs, local, noatime, nfsv4acls)
system/jails/cloud/home on /jails/cloud/home (zfs, local, noatime, nfsv4acls)
system/jails/cloud/tmp on /jails/cloud/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr on /jails/cloud/usr (zfs, local, noatime, nfsv4acls)
system/jails/cloud/usr/local on /jails/cloud/usr/local (zfs, local, noatime, nfsv4acls)
system/jails/cloud/usr/obj on /jails/cloud/usr/obj (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr/ports on /jails/cloud/usr/ports (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/usr/src on /jails/cloud/usr/src (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var on /jails/cloud/var (zfs, local, noatime, nfsv4acls)
system/jails/cloud/var/audit on /jails/cloud/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/crash on /jails/cloud/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/log on /jails/cloud/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/mail on /jails/cloud/var/mail (zfs, local, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/mysql on /jails/cloud/var/mysql (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/tmp on /jails/cloud/var/tmp (zfs, local, noatime, noexec, nosuid, nfsv4acls)
system/jails/cloud/var/www on /jails/cloud/var/www (zfs, local, noatime, noexec, nosuid, nfsv4acls)
storage on /storage (zfs, local, noatime, nfsv4acls)
storage/cloud on /storage/cloud (zfs, local, noatime, nfsv4acls)
storage/cloud/bc on /storage/cloud/bc (zfs, local, noatime, nfsv4acls)
storage/cloud/jocmei on /storage/cloud/jocmei (zfs, local, noatime, nfsv4acls)
/storage/cloud on /jails/cloud/storage (nullfs, local)
devfs on /jails/cloud/dev (devfs, local, multilabel)
 
Code:
storage on /storage (zfs, local, noatime, nfsv4acls)
storage/cloud on /storage/cloud (zfs, local, noatime, nfsv4acls)
storage/cloud/bc on /storage/cloud/bc (zfs, local, noatime, nfsv4acls)
storage/cloud/jocmei on /storage/cloud/jocmei (zfs, local, noatime, nfsv4acls)
/storage/cloud on /jails/cloud/storage (nullfs, local)
Ok, this looks good. But note that the jail doesn't have access to */cloud/bc and */cloud/jocmei, only */cloud/ itself is mounted on the jail.
 
Code:
storage on /storage (zfs, local, noatime, nfsv4acls)
storage/cloud on /storage/cloud (zfs, local, noatime, nfsv4acls)
storage/cloud/bc on /storage/cloud/bc (zfs, local, noatime, nfsv4acls)
storage/cloud/jocmei on /storage/cloud/jocmei (zfs, local, noatime, nfsv4acls)
/storage/cloud on /jails/cloud/storage (nullfs, local)
Ok, this looks good. But note that the jail doesn't have access to */cloud/bc and */cloud/jocmei, only */cloud/ itself is mounted on the jail.
What? Because the jail can most definitely access those datasets, the following output is from inside the jail:

(Note: The "xxxxxxxx" are some sensitive names that I removed for the purpose of posting here in the forum)

Code:
root@bc:~ # ls -la /storage/
total 26
drwxr-xr-x   4 root  wheel   4 Aug 13 18:38 .
drwxr-xr-x  20 root  wheel  25 Aug  8 02:40 ..
drwxrwx---   9 www   www    13 Aug 13 22:24 bc
drwxr-xr-x   2 www   www     2 Aug 13 18:38 jocmei
root@bc:~ # ls -la /storage/bc
total 127
drwxrwx---  9 www   www       13 Aug 13 22:24 .
drwxr-xr-x  4 root  wheel      4 Aug 13 18:38 ..
-rw-r--r--  1 root  wheel    324 Aug 13 03:27 .htaccess
-rw-r--r--  1 www   www        0 Aug 13 03:27 .ocdata
drwxr-xr-x  4 www   www        4 Aug 13 11:35 admin
drwxr-xr-x  9 www   www        9 Aug 13 12:02 appdata_yyyyyyyyy
drwxr-xr-x  7 www   www        7 Aug 13 20:40 xxxxxxxx
drwxr-xr-x  2 www   www        3 Aug 13 11:35 files_external
-rw-r--r--  1 www   www        0 Aug 13 03:27 index.html
drwxr-xr-x  4 www   www        4 Aug 13 14:12 xxxxxxxx
drwxr-xr-x  4 www   www        4 Aug 13 14:00 news
-rw-r-----  1 www   www    60837 Aug 14 11:14 nextcloud.log
drwxr-xr-x  6 www   www        6 Aug 13 22:30 xxxxxxxx
 
The /storage/bc and all are separate filesystems, so you want them available inside the jail you will have to nullfs(4) them separately.

Confusing but /jails/cloud/storage/bc refers to a subdirectory of /storage/ and not the /storage/bc filesystem.
 
The /storage/bc and all are separate filesystems, so you want them available inside the jail you will have to nullfs(4) them separately.

Confusing but /jails/cloud/storage/bc refers to a subdirectory of /storage/ and not the /storage/bc filesystem.
Ahh, now I get what was going on, nullfs does not operate across file systems! As you said, /jails/cloud/storage/bc was thus not pointing to the storage/cloud/bc file system, but instead a directory on the top-level file system, storage.

So what was happening inside the jail was that it was writing the data to that directory on storage, thus leaving the file system storage/cloud/bc empty. Meanwhile, outside the jail, the empty storage/cloud/bc file system was mounted over the directory on storage where the data actually resided and thereby "hiding" them from view outside the jail.

To anybody reading this in the future (Okay, that sounded more dramatic than it ought to! :D ):
If you are using ZFS and want to put a file system and all of its children into a jail then you cannot use nullfs(). Instead you need to either change the actual mount point using zfs set mountpoint=/jails/jail1/storage storage/path/to/filesystem or to attach the entire dataset to the jail, and allow it to be managed from within the jail using zfs jail.
 
Ahh, now I get what was going on, nullfs does not operate across file systems! As you said, /jails/cloud/storage/bc was thus not pointing to the storage/cloud/bc file system, but instead a directory on the top-level file system, storage.
Yes. That's what was happening. Mounts never cross filesystem boundaries. This isn't limited to nullfs(4) and/or ZFS, NFS and UFS for example 'suffer' from the same limitation. It's due to the way mounting filesystems work. Especially with multiple (separate) filesystems it's really easy to overlook this.
 
To anybody reading this in the future (Okay, that sounded more dramatic than it ought to! :D ):
If you are using ZFS and want to put a file system and all of its children into a jail then you cannot use nullfs(). Instead you need to either change the actual mount point using zfs set mountpoint=/jails/jail1/storage storage/path/to/filesystem or to attach the entire dataset to the jail, and allow it to be managed from within the jail using zfs jail.

A big THANK YOU from the "future" -

I ran into a very similar problem (taking me a full morning - longer if I hadn't read this). I nullfs mounted a "top-level" directory and hoping to access a subfolder, not awaring the subfolder is a separate ZFS dataset.

After reading this post, I explicitly nullfs-mounted the subfolder into my jail. I didn't use zfs mount because I need the dataset to be available at multiple mount points and zfs doesn't support that.
 
Yes. That's what was happening. Mounts never cross filesystem boundaries.

This is one of the few cases where Linux's version of a program is superior to FreeBSD's. Linux's mount() has the rbind flag which mounts submounts (crossing filesystems):

The bind mount call attaches only (part of) a single filesystem, not possible submounts. The entire file hierarchy including submounts is attached a second place using

mount --rbind olddir newdir

or shortoption

mount -R olddir newdir
 
  • Thanks
Reactions: klu
Another thank you from the future :D
I tried mounting several ZFS datasets through nullfs on a UFS top-level media and then nullfs-mount that UFS media into the jail (all done from fstab.jailname). This obviously didn't work as expected leading me to find this thread and rethinking the topography of the mounts...
 
Back
Top