Other [Solved] Nullfs sometimes fail to mount on FreeBSD 12.1

U

Unga

Dear all

I mount GlusterFS at boot and use nullfs to remount part of GlusterFS to somewhere else, both via /etc/fstab. This remount “sometime” fails at boot. That is, it works too sometimes.

I’m running 12.1-RELEASE (amd64 on intel i7) and glusterfs-3.11.1_6.

My /etc/fstab:
Code: 
# Device Mountpoint FStype Options Dump Pass#
/dev/ada0p2 none swap sw 0 0
/dev/ada0p3 / ufs rw 1 1

# Mount GlusterFS volume
host1:/GFSVol /mnt/gluster/ fusefs rw,acl,transport=tcp,_netdev,backup-volfile-servers=host2:host3,mountprog=/usr/local/sbin/mount_glusterfs,late,failok,log-level=WARNING 0 0

# Mount home from GlusterFS
/mnt/gluster/Home      /home    nullfs    rw,late,failok  0  0

1. GlusterFS “always” get mounted on /mnt/gluster/.

2. The /mnt/gluster/Home “sometime” fails to mount on /home with following message on /var/log/messages:
Code: 
kernel: mount_nullfs: /mnt/gluster/Home: Socket is not connected
Am I missing any required mount option?

Symptom shows that /mnt/gluster/Home trying to get mounted before /mnt/gluster get mounted.

What I miss in my mount options is a way to declare that /mnt/gluster is required for /mnt/gluster/Home mounting. Eg. require="/mnt/gluster". Is there such an option in FreeBSD?

Thanks.
 
This has nothing to do with your issue, it's just an observation. But /mnt is meant to be used as a temporary mount point, not a permanent one. I often see people more or less abusing it for (semi-) permanent mounts. Just create a directory specific for your (external) mounts, like /storage or /data.

Code: 
     /mnt/      empty directory commonly used by system administrators as a
                temporary mount point
See hier(7)
 
1. Thank you SirDice. Well noted.

2. I'm beginning to feel my issue may be due to order of late mounts are carried out. Probably late mounts are carried out in random order, not in the order mounts are specified in /etc/fstab.
 
Why use nullfs(5)? Why not simply a symlink from /home to where GlusterFS is mounted? Have you tried removing the late option on the GlusterFS mount?
 
1. Why use nullfs(5)?
MOUNT_NULLFS(8)
"The primary differences between a virtual copy of the file system and a symbolic link are that the getcwd(3) functions work correctly in the virtual copy,...".

2. How I do the same on Ubuntu Linux 18.04 LTS (/etc/fstab):
/mnt/gluster/Home /home none defaults,bind,_netdev,nofail 0 0

On Linux this does not fail at all. I cannot remember a single failure.

3. Have you tried removing the late option on the GlusterFS mount?
Yes, FreeBSD does not boot. Hangs after non-late file systems are mounted from /etc/fstab.

4. Another possibility for failure is nullfs may be getting mounted before the networking is up. FreeBSD error message is "Socket is not connected". Note, on Linux "_netdev" is specified to mount after networking is up.

I cannot remember whether I tried "_netdev" with nullfs. I'll try later and update here.

Pls see: http://manpages.ubuntu.com/manpages/bionic/man8/mount.8.html
 
1. I tried "_netdev" with nullfs. Four times rebooted, all four times /mnt/gluster/Home got mounted to /home. Is it just random?

2. Where is "_netdev" implemented in FreeBSD source? It seems "_netdev" is not documented anywhere in FreeBSD.
 
Ok removed "_netdev" from fstab. Rebooted. Mounting seems ok. Let me observe for about a day and wrap this up. Thank you.
 
1. The mounting issue did NOT solve. It happened again.
(/var/log/messages)
Jan 17 11:58:54 myhost kernel: mount_nullfs: /mnt/gluster/Home: Socket is not connected

2. My /etc/fstab:
/dev/ada0p2 none swap sw 0 0
/dev/ada0p3 / ufs rw 1 1

# Mount GlusterFS volume
host1:/GFSVol /mnt/gluster/ fusefs ...,late,failok,... 0 0

# Mount home from GlusterFS
/mnt/gluster/Home /home nullfs rw,late,failok 0 0

3. Do late mount entries in /etc/fstab mounted in the order specified in /etc/fstab? Or how to enforce the order?
 
Unga said:
Do late mount entries in /etc/fstab mounted in the order specified in /etc/fstab?
Click to expand...
Yes.

I suspect it's simply because GlusterFS takes a little while to actually mount and the nullfs(5) gets mounted before GlusterFS finishes.

You could perhaps use autofs(5) for the nullfs(5) mount, then it gets mounted when it's actually needed. In the mean time GlusterFS gets some extra time to initialize and mount its filesystems.
 
I'm not entirely sure, but this might concern your situation. Have a look at this open PR 236112, with "Summary: net/glusterfs: Startup should happen before mountlate", solution included.
 
1. Yes, it's entirely required. I'm already using. GlusterFS must be up before mountlate starts.

2. Someone who has enough permissions please effect PR 236112 to /usr/local/etc/rc.d/glusterd. It's nearly a year old now. Things should not miss like this.

3. Issue in FreeBSD now clear, with PR 236112 in effect, mountlate only happens after GlusterFS is up. That's why my mounting GlusterFS succeeds.

But the issue is, the mount program either continue to process the next mount entry in fstab without either checking the status of the current mount entry or mount_glusterfs returns success but the mount program proceeds to next fstab entry before the mountpoint (/mnt/gluster/) is ready and available for use. (I did not check the mount program source yet, bit of fighting with time now).

In my opinion, it's the responsibility of the mount program (the program processes the fstab) to make sure the mountpoint is ready for use within a timeout period. If the mountpoint is not ready before timeout, it must make an error message and proceed to next fstab entry. In this case, a mount option "timeout" is required for users to increase the timeout if required.

4. Till then, I'm going to try mount part of the GlusterFS via nullfs by a rc script, so that I can push the processing this rc script to possibly end of rc processing.
 
Solved.

1. Mount part of the GlusterFS via nullfs by a rc script works well.
2. For anyone interested, order of rc scripts as follows:
/usr/local/etc/rc.d/glusterd
/etc/rc.d/mountlate
/usr/local/etc/rc.d/mounthome
 
You must log in or register to reply here.
Back
Top