Solved NFS not mounting on FreeBSD on boot

V

VulcanRidr

I have noticed in the past month or two that on my desktop and laptop, both of which are currently running 13.0-RELEASE-p1, the NFS mounts from my TrueNAS-13u5.2 host is no longer being mounted on boot. Unfortunately, I cannot pinpoint exactly when it stopped mounting at boot, but I can go in after boot and do a mount -a, but during boot it's not happening. I also, though I have never had to do it before, went in and added nfs_client_enable="YES" to rc.conf, which I have never had to do before, and the datasets still didn't mount at boot.

I tried several different boot environments, going back to the oldest I have on this machine, which is 13.1p7 on the FreeBSD desktop. All failed. I also, since I dual boot that box with Devuan linux, booted into that, and the NFS mounts mounted on boot. I also checked the TrueNAS box, Sharing -> NFS, Services -> NFS -> config (all boxes unchecked).

Did something change or am I missing something?

Thanks,
--vr
 
Show us the contents of /etc/exports on the NFS server and, on the client grep -i nfs /etc/rc.conf /etc/fstab
 
gpw928 said:
Show us the contents of /etc/exports on the NFS server and, on the client grep -i nfs /etc/rc.conf /etc/fstab
Click to expand...
Code: 
/mnt/NX80101/backups -maproot="user":"user" -network 192.168.10.0/24
/mnt/NX80101/audio_clips 192.168.10.1 192.168.10.5 172.20.10.2
/mnt/NX80101/library defiant danube valiant
/mnt/NX80101/iso -alldirs defiant danube intrepid
/mnt/NX80101/video_clips -alldirs 192.168.10.1 192.168.10.5 172.20.10.2
/mnt/NX80101/media -alldirs defiant danube valiant
/mnt/NX80101/games defiant danube 172.20.10.2

And the fstab on this host:

Code: 
/etc/rc.conf:nfs_client_enable="YES"
/etc/fstab:luna:/mnt/NX80101/media/video /exports/video nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/media/music /exports/music nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/iso   /exports/iso   nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/games   /exports/games   nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/video_clips /exports/extras/video_clips nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/audio_clips /exports/extras/audio_clips nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/library /exports/library nfs rw 0 0
/etc/fstab:luna:/mnt/NX80101/backups /exports/backups nfs rw 0 0
 
We also need to know the name and IP address of one of the FreeBSD NFS clients that is not doing the NFS mounts at boot time.

Edit: plus the IP address(es) of the NFS server, and probably the output of netstat -r -n on that server
 
So you are saying that:
  • You have NFS file systems that are being exported by a server.
  • You can mount them from your FreeBSD client, without any problem, after the client has booted.
  • Yet mounting them automatically at boot time does not work.
  • But, most importantly, that automatic boot used to work, and is now broken.
Here would be my suggestion (after you're done working through the homework that gpw928 gave you): Can you pinpoint when it last worked, and what you have changed since then? And how much is your FreeBSD install modified? For example, have you been editing the /etc/rc.d/... files, and changed their prerequisites, so they are being run in the wrong order? How does your client get its IP address, and has that changed? Have you been adjusting how DHCP and DNS work? Or installed or administered some firewalls (that could be PF, in software)?
 
gpw928 said:
We also need to know the name and IP address of one of the FreeBSD NFS clients that is not doing the NFS mounts at boot time.
Click to expand...
Sorry, was late. Both defiant and danube are having this problem. Both are 13.2p1, though the problem occurs on older releases (I booted into 13.1p7 to test).
gpw928 said:
Edit: plus the IP address(es) of the NFS server, and probably the output of netstat -r -n on that server
Click to expand...
The NFS server is on 192.168.10.20. Same subnet. And here is the routing table on the TrueNAS:
Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.10.4       UGS        igb0
127.0.0.1          link#3             UH          lo0
192.168.10.0/24    link#1             U          igb0
192.168.10.20      link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
 
ralphbsz said:
So you are saying that:
  • You have NFS file systems that are being exported by a server.
  • You can mount them from your FreeBSD client, without any problem, after the client has booted.
  • Yet mounting them automatically at boot time does not work.
  • But, most importantly, that automatic boot used to work, and is now broken.
Click to expand...
This is essentially correct.
ralphbsz said:
Here would be my suggestion (after you're done working through the homework that gpw928 gave you): Can you pinpoint when it last worked, and what you have changed since then? And how much is your FreeBSD install modified? For example, have you been editing the /etc/rc.d/... files, and changed their prerequisites, so they are being run in the wrong order? How does your client get its IP address, and has that changed? Have you been adjusting how DHCP and DNS work? Or installed or administered some firewalls (that could be PF, in software)?
Click to expand...
As for when it started, it was, maybe in the May/June 2023 timeframe? I haven't made a lot of changes to either box, they are my daily drivers. About the only "changes" I make to them are to freebsd-update fetch install and pkg upgrade ("current" repo) them on a fairly regular basis. I have kept abreast of the base system changes (13.1p7 -> 13.2 -> 13.2p1), and try to keep up with package releases (pkg upgrades every week to two weeks). Just updated TrueNAS from the 13.0u5.1 to 13.0u5.2 yesterday. No changes (unless they were done by freebsd-update or pkg upgrade) to the rc files, no changes to DHCP or DNS, and no firewall between them.

My clients get their IP addresses via DHCP, however, I have all of their MAC addresses mapped to IPs in the dhcp config. So defiant is always 192.168.10.1.
 
VucanRidr said:
My clients get their IP addresses via DHCP
Click to expand...
Try SYNCDHCP instead. You could also try to add late to the NFS mounts in fstab, that will cause them to be mounted a little later during the boot process.

Alternatively, you may want to consider moving your NFS mounts to autofs(8)/automount(8). If you reboot the system and your NFS shares aren't available the system will croak and get stuck in single user mode.

And, please, don't mount anything permanent on or under /mnt. It's meant for temporary mounts.

Code: 
     /mnt/      empty directory commonly used by system administrators as a
                temporary mount point
hier(7)
 
All right, the strangest thing just happened. On the desktop, I changed the first two entries from
Code: 
luna:/mnt/NX80101/media/video /exports/video nfs rw 0 0
luna:/mnt/NX80101/media/music /exports/music nfs rw 0 0
to
Code: 
luna:/mnt/NX80101/media/video /exports/video nfs rw,late 0 0
luna:/mnt/NX80101/media/music /exports/music nfs rw,late 0 0

Then I rebooted, and got the attached snap of my console. I went into single user mode, and removed the ",late" from the two entries above and rebooted. Ironically enough, it mounted the NFS from the NAS. I rebooted again, and it did not mount. I went into fstab and added ",late" to the two entries, got thrown into single user, removed the late entry, and rebooted again. This time, it did not mount from the NAS. It almost feels like adding late entries to the fstab is delaying the network start, though that doesn't make any sense to me. Should I add ",bg" to the late option?
 

Attachments

  • 20230721_003.jpg
    20230721_003.jpg
    602.1 KB · Views: 106
First take is that your name server is clearly not working at the time you are trying to do the NFS mount. To that end, adding the NFS server name and IP address into /etc/hosts on the clients is likely to help. If that fixes the problem, we can delve into the root cause.
 
How about late,failok in /etc/fstab and use netwait to wait for IP to come up. You could use an interface based approach too.

/etc/rc.conf
netwait_enable="YES" # Enable rc.d/netwait (or NO)
netwait_ip="192.168.10.1" # Wait for ping response from any IP in this list.
netwait_timeout="60" # Total number of seconds to perform pings.
 
Phishfry said:
Code: 
netwait_enable="YES"        # Enable rc.d/netwait (or NO)
netwait_ip="192.168.10.1"            # Wait for ping response from any IP in this list.
netwait_timeout="60"        # Total number of seconds to perform pings.
Click to expand...
That's likely to work (the NFS server at 192.168.10.20 would be a better target), but doing that won't tell us why.
The underlying cause may be a name server, a network interface, or something else. I expect that rcorder(8) may eventually tell us why.
 
Sorry to interrupt you as I just see so many cases where netwait can help.

Above I was just grabbing a server IP there from a post. Unsure if correct.
A living example if you will.

The OP mentions he uses DHCP but static IP's.
So an interface based netwait or IP based netwait is a choice worth considering. Either should work.
(protoypes found in /etc/defaults/rc.conf)

I like the nofail option in /etc/fstab at least until you get everything worked out.
But I would not recommend SYNCDHCP as best option here contrary to my mentor mister SirDice .
 
Phishfry said:
Sorry to interrupt you as I just see so many cases where netwait can help.

Above I was just grabbing a server IP there from a post. Unsure if correct.
A living example if you will.

The OP mentions he uses DHCP but static IP's.
So an interface based netwait or IP based netwait is a choice worth considering. Either should work.
(protoypes found in /etc/defaults/rc.conf)

I like the nofail option in /etc/fstab at least until you get everything worked out.
But I would not recommend SYNCDHCP as best option here contrary to my mentor mister SirDice .
Click to expand...
I understand what you are saying here, and to a large degree, I agree with you. That said, I would also, as gpw928 said, like to know what caused the behavior to change...Unfortunately, I am still relatively new to FreeBSD, which is why I called for help. I had a friend suggest just sticking a script in to do the mount -a after boot, but I told him that that is the kind of script that gets "lost" in the mists of time and then when it breaks, you spend all sorts of time banging your head on your keyboard trying to figure out why it is broken.
 
Phishfry said:
How about late,failok in /etc/fstab and use netwait to wait for IP to come up. You could use an interface based approach too.

/etc/rc.conf
netwait_enable="YES" # Enable rc.d/netwait (or NO)
netwait_ip="192.168.10.1" # Wait for ping response from any IP in this list.
netwait_timeout="60" # Total number of seconds to perform pings.
Click to expand...
I think I had a breakthrough. I tried the netwait enable (which worked) on the laptop. Then, to make sure that it wasn't a problem with the DNS as was suggested by ralphbsz, so I disabled netwait in /etc/rc.conf, and changed the /etc/fstab to point to the IP address instead of the hostname. When I rebooted, I got a bunch of IPv6 stuff, and then the mounting of the NFS partitions timed out. I am not using IPv6, so I added p6addrctl_enable="NO" to rc.conf, rebooted, and NFS came up instantly.

Changed the fstab back to hostname, and it still worked...So apparently it was something getting confused with inet6.

Thoughts?
 
There's too many unknowns to be sure. We would need to look at the IPV6 stacks on clients and server, plus consider packet filters.

As Phishfry suggested, the netwait option covers a multitude of network co-ordination issues, provided you wait on the appropriate interfaces.
 
gpw928 said:
There's too many unknowns to be sure. We would need to look at the IPV6 stacks on clients and server, plus consider packet filters.

As Phishfry suggested, the netwait option covers a multitude of network co-ordination issues, provided you wait on the appropriate interfaces.
Click to expand...
That is what I did on both the laptop and the desktop. Disabled ipv6, and set up netwait. So I added the following to the end of rc.conf:

Code: 
ipv6_network_interfaces="none"
ip6addrctl_enable="NO"
netwait_enable="YES"
netwait_ip="192.168.10.20"
netwait_timeout="60"

So while I never figured out what caused the problem, the netwait seems to have resolved it, so I am tentatively marking this solved. I thank every respondent for their helpful suggestions that pointed me where I could look.
 
You must log in or register to reply here.
Back
Top