ZFS mounted directory empty - full after remount

S

Simon Wagner

Hi,

my server (12.0-p9) developed a new quirk, it must have happened during the last 7 days. A dataset is mounted, but does not contain the expected data. After umount/mount, the data is there. After running out of ideas what I could ask google, I'm turning to you, dear Forum.

Below you see what I can do to recreate the error. These actions are fresh after a reboot.

Thanks a lot for pointing me in the right direction
Regards
simon

Code: 
root@psrPLGRND001:~ # zfs get canmount,mountpoint laser/VirtualBox_VMs                                                                                                                                       
NAME                  PROPERTY    VALUE                      SOURCE
laser/VirtualBox_VMs  canmount    on                         default
laser/VirtualBox_VMs  mountpoint  /home/vbox/VirtualBox_VMs  local

root@hostname:~ # zfs mount | grep VirtualBox_VMs
laser/VirtualBox_VMs            /usr/home/vbox/VirtualBox_VMs

root@hostname:~ # ls -l /usr/home/vbox/VirtualBox_VMs
total 0

root@hostname:~ # zfs umount laser/VirtualBox_VMs                                                                                                                                                         

root@hostname:~ # ls -l /usr/home/vbox/VirtualBox_VMs                                                                                                                                                     
total 0

root@hostname:~ # zfs mount laser/VirtualBox_VMs     

root@hostname:~ # ls -l /usr/home/vbox/VirtualBox_VMs                                                                                                                                                     
total 448937
drwx------  2 vbox  vboxusers          3 Jun 26 16:33 One
drwx------  4 vbox  vboxusers          7 Aug  6 10:16 Two
drwxr-xr-x  2 root  vboxusers          4 Jun 26 16:38 Three
root@hostname:~ #
 
Check the full mount list, I have a feeling you might be mounting two separate filesystems on that directory. Order becomes important then. By remounting you're effectively changing the order.
 
It gets odder:

after unmounting the 'empty' mount, there was still an (empty) directory with that name. So I rm'd it.

Now, when I reboot, the directory is shown in 'zfs mount', but not in the ls. After un/remounting, everything is there....

I see no double mounts or underlying/masking directories.

Code: 
root@psrPLGRND001:~ # cd /usr/home/vbox/
root@psrPLGRND001:/usr/home/vbox # ls -l
total 33
drwx------  3 vbox  vboxusers     3 Jun 26 15:10 .config
-rw-r--r--  1 vbox  vboxusers  1054 Jun 26 15:10 .cshrc
-rw-------  1 vbox  vboxusers    51 Aug 20 10:32 .history
-rw-r--r--  1 vbox  vboxusers   392 Jun 26 15:10 .login
-rw-r--r--  1 vbox  vboxusers   163 Jun 26 15:10 .login_conf
-rw-------  1 vbox  vboxusers   379 Jun 26 15:10 .mail_aliases
-rw-r--r--  1 vbox  vboxusers   339 Jun 26 15:10 .mailrc
-rw-r--r--  1 vbox  vboxusers   954 Jun 26 15:10 .profile
-rw-r--r--  1 vbox  vboxusers   851 Jun 26 15:10 .shrc
drwxr-xr-x  4 root  vboxusers     4 Aug 20 14:32 OLD
root@psrPLGRND001:/usr/home/vbox # zfs mount
zroot/ROOT/default              /
laser/VirtualBox_VMs            /usr/home/vbox/VirtualBox_VMs
laser                           /mnt/laser
zroot/tmp                       /tmp
zroot/usr/home                  /usr/home
zroot/usr/ports                 /usr/ports
zroot/usr/src                   /usr/src
zroot/var/audit                 /var/audit
zroot/var/crash                 /var/crash
zroot/var/log                   /var/log
zroot/var/mail                  /var/mail
zroot/var/tmp                   /var/tmp
zroot                           /zroot
root@psrPLGRND001:/usr/home/vbox # zfs umount laser/VirtualBox_VMs
root@psrPLGRND001:/usr/home/vbox # zfs mount laser/VirtualBox_VMs
root@psrPLGRND001:/usr/home/vbox # ls -l
total 42
drwx------   3 vbox  vboxusers     3 Jun 26 15:10 .config
-rw-r--r--   1 vbox  vboxusers  1054 Jun 26 15:10 .cshrc
-rw-------   1 vbox  vboxusers    51 Aug 20 10:32 .history
-rw-r--r--   1 vbox  vboxusers   392 Jun 26 15:10 .login
-rw-r--r--   1 vbox  vboxusers   163 Jun 26 15:10 .login_conf
-rw-------   1 vbox  vboxusers   379 Jun 26 15:10 .mail_aliases
-rw-r--r--   1 vbox  vboxusers   339 Jun 26 15:10 .mailrc
-rw-r--r--   1 vbox  vboxusers   954 Jun 26 15:10 .profile
-rw-r--r--   1 vbox  vboxusers   851 Jun 26 15:10 .shrc
drwxr-xr-x   4 root  vboxusers     4 Aug 20 14:32 OLD
drwxr-xr-x  10 vbox  vboxusers    11 Jul  1 10:17 VirtualBox_VMs
root@psrPLGRND001:/usr/home/vbox #
 
Simon Wagner said:
after unmounting the 'empty' mount, there was still an (empty) directory with that name.
Click to expand...
Yes, it's supposed to, that's your mount-point. A mount-point needs to exist before you can mount a filesystem on it. Try mounting a filesystem on a non-existing directory to see what I mean.

Simon Wagner said:
I see no double mounts or underlying/masking directories.
Click to expand...
You've only checked ZFS mounts. It might be UFS, tmpfs(5) or a mdmfs(8). Look at the output from mount. Compare before and after you remounted that laser/VirtualBox_VMs filesystem.
 
Ok, cranial diarrhea on my part...

the empty directory/mount point is back after umount/mouting, and I'm leaving it alone this time ;-)

There IS some crazy overlay happening, especially when the first umount is possible while I'm in the directory.

I just don't see, where it comes from?

Code: 
root@psrPLGRND001:~vbox # ls -l
total 34
drwx------  3 vbox  vboxusers     3 Jun 26 15:10 .config
-rw-r--r--  1 vbox  vboxusers  1054 Jun 26 15:10 .cshrc
-rw-------  1 vbox  vboxusers    51 Aug 20 10:32 .history
-rw-r--r--  1 vbox  vboxusers   392 Jun 26 15:10 .login
-rw-r--r--  1 vbox  vboxusers   163 Jun 26 15:10 .login_conf
-rw-------  1 vbox  vboxusers   379 Jun 26 15:10 .mail_aliases
-rw-r--r--  1 vbox  vboxusers   339 Jun 26 15:10 .mailrc
-rw-r--r--  1 vbox  vboxusers   954 Jun 26 15:10 .profile
-rw-r--r--  1 vbox  vboxusers   851 Jun 26 15:10 .shrc
drwxr-xr-x  4 root  vboxusers     4 Aug 20 14:32 OLD
drwxr-xr-x  2 root  vboxusers     2 Aug 20 14:36 VirtualBox_VMs
root@psrPLGRND001:~vbox # mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
laser/VirtualBox_VMs on /usr/home/vbox/VirtualBox_VMs (zfs, local, noatime, nfsv4acls)
laser on /mnt/laser (zfs, local, noatime, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
root@psrPLGRND001:~vbox # zfs mount
zroot/ROOT/default              /
laser/VirtualBox_VMs            /usr/home/vbox/VirtualBox_VMs
laser                           /mnt/laser
zroot/tmp                       /tmp
zroot/usr/home                  /usr/home
zroot/usr/ports                 /usr/ports
zroot/usr/src                   /usr/src
zroot/var/audit                 /var/audit
zroot/var/crash                 /var/crash
zroot/var/log                   /var/log
zroot/var/mail                  /var/mail
zroot/var/tmp                   /var/tmp
zroot                           /zroot
root@psrPLGRND001:~vbox # cd VirtualBox_VMs/
root@psrPLGRND001:~vbox/VirtualBox_VMs # ls -l
total 0
root@psrPLGRND001:~vbox/VirtualBox_VMs # touch test_file
root@psrPLGRND001:~vbox/VirtualBox_VMs # zfs umount laser/VirtualBox_VMs
root@psrPLGRND001:~vbox/VirtualBox_VMs # ls -l
total 1
-rw-r--r--  1 root  vboxusers  0 Aug 20 16:39 test_file
root@psrPLGRND001:~vbox/VirtualBox_VMs # zfs mount laser/VirtualBox_VMs 
root@psrPLGRND001:~vbox/VirtualBox_VMs # ls -l
total 448937
drwx------  2 vbox  vboxusers          3 Jun 26 16:33 BackupTest
drwx------  4 vbox  vboxusers          7 Aug 20 16:30 LDAP
drwxr-xr-x  2 root  vboxusers          4 Jun 26 16:38 installers
drwx------  2 vbox  vboxusers          4 Jun 26 16:33 micro
drwx------  3 vbox  vboxusers          6 Jun 26 16:33 microMachine
-rw-------  1 vbox  vboxusers  459787776 Jun 26 16:36 microMachine.ova
drwx------  3 vbox  vboxusers          6 Jul 18 12:39 oracle Test
drwx------  4 vbox  vboxusers          7 Jul  1 10:17 small
drwx------  3 vbox  vboxusers          6 Jun 26 16:36 usrlocal_test
root@psrPLGRND001:~vbox/VirtualBox_VMs # zfs umount laser/VirtualBox_VMs
cannot unmount '/usr/home/vbox/VirtualBox_VMs': Device busy
root@psrPLGRND001:~vbox/VirtualBox_VMs # mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
laser on /mnt/laser (zfs, local, noatime, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
laser/VirtualBox_VMs on /usr/home/vbox/VirtualBox_VMs (zfs, local, noatime, nfsv4acls)
root@psrPLGRND001:~vbox/VirtualBox_VMs # zfs mount
zroot/ROOT/default              /
laser                           /mnt/laser
zroot/tmp                       /tmp
zroot/usr/home                  /usr/home
zroot/usr/ports                 /usr/ports
zroot/usr/src                   /usr/src
zroot/var/audit                 /var/audit
zroot/var/crash                 /var/crash
zroot/var/log                   /var/log
zroot/var/mail                  /var/mail
zroot/var/tmp                   /var/tmp
zroot                           /zroot
laser/VirtualBox_VMs            /usr/home/vbox/VirtualBox_VMs
root@psrPLGRND001:~vbox/VirtualBox_VMs #
 
Code: 
root@psrPLGRND001:~vbox # mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
laser/VirtualBox_VMs on /usr/home/vbox/VirtualBox_VMs (zfs, local, noatime, nfsv4acls)
laser on /mnt/laser (zfs, local, noatime, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
Looks like zroot/usr/home is mounted after laser/VirtualBox_VMs. That would certainly mask it. As far as I understood things ZFS does some automatic sorting in order to mount filesystems in the correct order. But this ordering only works within the same pool and goes out the window when two (or more) separate pools are involved. Judging by your mount list I suspect the pools are ordered alphabetically, so filesystems on the laser pool are mounted before filesystems on the zroot pool.

In your case I would probably use a /storage/VirtualBox_VMs mount point and simply use a symlink from /usr/home/vbox/VirtualBox_VMs to /storage/VirtualBox_VMs.
 
Your explanation makes sense. I found another thread that touches the same subject.

Symlink is likely the best short-term solution, just to get things working.
 
Simon Wagner said:
Symlink is likely the best short-term solution, just to get things working.
Click to expand...
It might even be the only solution, I have not been able to find anything regarding the mount order from different pools. No way to change the order it appears. Alternatively you could fallback to legacy and use fstab for this. That will definitely allow you to change the mount order.
 
You must log in or register to reply here.
Back
Top