RPC: Port mapper failure

G

Grazzy

We have a physical machine running FreeBSD 10.1. It is setup as an nfs server for multiple VM servers acting as nfs clients. The machine has been stable for a long time.

I was ssh logged into nfs machine. I did the following.
freebsd-update fetch
Ctrl+C (after a few sec because I realized I didn't want to sit through this process)
shutdown -r now

The machine did not reboot.
I went to the physical machine and did a hard reboot. It hangs in a loop.
Code: 
(tcp) nfs.xxx.xxx:/zraid10/FreeBSD: RPCPROG_NFS: RPC: Port mapper failure - RPC: Timed out

Q. Did my bailing out of the FreeBSD update fetch cause this?

Q. What is my path forward to fixing this?

Thanks for any input.
 
Grazzy said:
Q. Did my bailing out of the FreeBSD update fetch cause this?
Click to expand...

A freebsd-update fetch never got the chance to modify any in use files. Only an interrupted freebsd-update install would cause a potential issue.

Grazzy said:
Q. What is my path forward to fixing this?
Click to expand...

Boot in single user mode and comment out all NFS mounts from /etc/fstab. Do return to let it boot fully. Then troubleshoot why it's having issues mounting.
 
Thanks junovitch@ for the response!

junovitch@ said:
A freebsd-update fetch never got the chance to modify any in use files. Only an interrupted freebsd-update install would cause a potential issue.
Click to expand...
I guess I kind of knew that but it helps to get confirmation. Thanks.

junovitch@ said:
Boot in single user mode and comment out all NFS mounts from /etc/fstab. Do return to let it boot fully. Then troubleshoot why it's having issues mounting.
Click to expand...

I booted to Single-user Mode and attempted to modify fstab.
Unfortunately I am unable to modify the file.
vim /etc/fstab

When I insert the # char I get:
Code: 
E303: Unable to open swap file for "etc/fstab", recovery impossible
The line does get commented out though:
Code: 
:wq!

  "/etc/fstab" E212: Can't open file for writing

:q!

  E138: Can't write viminfo file /root/.viminfo

Here is the contents of my fstab file.
Code: 
Device    Mountpoint  FStype  Options  Dump  Pass#
/dev/gpt/root  /  usf  rw  1  1
/dev/gpt/swap  none  swap  sw  0  0
nfs.xxx.xxx:/zraid10/freebsd  /freebsd  nfs  ro,intr,nosuid  0  0

I appreciate any continuing suggestions.

Update: I was able to edit fstab by first entering mount -uw /. I'll see what happens from this point.
 
Last edited by a moderator:
OK my NFS machine is back up and running.
In Single-user Mode I commented out the NFS mount line and continued the boot.
It took me to the login prompt and everything appears to be back to normal.
I checked fstab and the line was restored at some point during the final boot process.
So now when I restart the machine I have to comment out the line first and then the boot proceeds normally.

I believe the root cause is an issue with the order in the boot init routine. That will have to be investigated. But for now at least I know how to work around it.

Thank you very much junovitch@. You rock!
 
Does adding the 'late' option to /etc/fstab for that mount help? I don't see that as a recommendation in the FreeBSD Handbook: NFS section but it may be an option. There may be some local network configuration to consider that is preventing this from working at boot. What about name resolution not being available when /etc/rc.d/mountcritremote runs?
 
junovitch@ said:
Does adding the 'late' option to /etc/fstab for that mount help? I don't see that as a recommendation in the FreeBSD Handbook: NFS section but it may be an option. There may be some local network configuration to consider that is preventing this from working at boot. What about name resolution not being available when /etc/rc.d/mountcritremote runs?
Click to expand...

Thanks for the additional suggestions. It may be a while before I can this investigate further. With the machine running and a work-around in place to allow a server restart, root cause gets bumped below other higher priorities.
 
Since you are using DNS names it seems like a logical possibility. What DNS software are you using? Is it listed before /etc/rc.d/mountcritremote in rcorder /etc/rc.d/* /usr/local/etc/rc.d/*.
 
junovitch@ said:
What DNS software are you using?
Click to expand...
I believe that would be BIND on a FreeBSD name server VM.

junovitch@ said:
Is it listed before /etc/rc.d/mountcritremote in rcorder /etc/rc.d/* /usr/local/etc/rc.d/*.
Click to expand...
Here are the results of running
root@nfs:/ # rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
mountcritlocal is listed prior to anything with "bind" in it. I see rpcbind and ybind.
Code: 
rcorder: file `/usr/local/etc/rc.d/tcsd' is before unknown provision `kerberos'
rcorder: file `/usr/local/etc/rc.d/tcsd' is before unknown provision `named'
rcorder: requirement `tpmd' in file `/usr/local/etc/rc.d/tcsd' has no providers.
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/initrandom
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap
/etc/rc.d/fsck
/etc/rc.d/root
/etc/rc.d/mdconfig
/etc/rc.d/hostid_save
/etc/rc.d/mountcritlocal
/etc/rc.d/zfs
/etc/rc.d/var
/etc/rc.d/cleanvar
/etc/rc.d/FILESYSTEMS
/etc/rc.d/kldxref
/etc/rc.d/kld
/etc/rc.d/addswap
/etc/rc.d/random
/etc/rc.d/postrandom
/etc/rc.d/adjkerntz
/etc/rc.d/atm1
/etc/rc.d/hostname
/etc/rc.d/ip6addrctl
/etc/rc.d/netoptions
/etc/rc.d/sppp
/etc/rc.d/ipfilter
/etc/rc.d/ipnat
/etc/rc.d/ipfs
/etc/rc.d/serial
/etc/rc.d/netif
/etc/rc.d/devd
/etc/rc.d/ipsec
/etc/rc.d/atm2
/etc/rc.d/pfsync
/etc/rc.d/pflog
/etc/rc.d/pf
/etc/rc.d/stf
/etc/rc.d/ppp
/etc/rc.d/faith
/etc/rc.d/routing
/etc/rc.d/mroute6d
/etc/rc.d/nsswitch
/etc/rc.d/rtsold
/etc/rc.d/resolv
/etc/rc.d/local_unbound
/etc/rc.d/static_ndp
/etc/rc.d/static_arp
/etc/rc.d/bridge
/etc/rc.d/route6d
/etc/rc.d/mrouted
/etc/rc.d/routed
/etc/rc.d/defaultroute
/etc/rc.d/ipfw
/etc/rc.d/NETWORKING
/etc/rc.d/netwait
/etc/rc.d/mountcritremote
/etc/rc.d/devfs
/etc/rc.d/ipmon
/etc/rc.d/mdconfig2
/etc/rc.d/ldconfig
/usr/local/etc/rc.d/nslcd
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/kdc
/etc/rc.d/watchdogd
/etc/rc.d/savecore
/etc/rc.d/archdep
/etc/rc.d/abi
/etc/rc.d/SERVERS
/usr/local/etc/rc.d/tcsd
rcorder: requirement `cupsd' in file `/usr/local/etc/rc.d/samba' has no providers.
/etc/rc.d/accounting
/etc/rc.d/ntpdate
/etc/rc.d/rpcbind
/etc/rc.d/nfsclient
/etc/rc.d/nisdomain
/etc/rc.d/ypserv
/etc/rc.d/ypbind
/etc/rc.d/ypset
/etc/rc.d/amd
/etc/rc.d/atm3
/etc/rc.d/audit
/etc/rc.d/auditdistd
/etc/rc.d/tmp
/etc/rc.d/cleartmp
/etc/rc.d/ctld
/etc/rc.d/dmesg
/etc/rc.d/hastd
/etc/rc.d/ipxrouted
/etc/rc.d/iscsid
/etc/rc.d/iscsictl
/etc/rc.d/keyserv
/etc/rc.d/nfsuserd
/etc/rc.d/gssd
/etc/rc.d/quota
/etc/rc.d/mountd
/etc/rc.d/nfsd
/etc/rc.d/statd
/etc/rc.d/lockd
/etc/rc.d/pppoed
/etc/rc.d/pwcheck
/etc/rc.d/virecover
/etc/rc.d/DAEMON
/usr/local/etc/rc.d/samba
/etc/rc.d/apm
/etc/rc.d/apmd
/etc/rc.d/bootparams
/etc/rc.d/hcsecd
/etc/rc.d/bthidd
/etc/rc.d/local
/etc/rc.d/lpd
/etc/rc.d/motd
/etc/rc.d/mountlate
/etc/rc.d/nscd
/etc/rc.d/ntpd
/etc/rc.d/powerd
/etc/rc.d/rarpd
/etc/rc.d/rctl
/etc/rc.d/sdpd
/etc/rc.d/rfcomm_pppd_server
/etc/rc.d/rtadvd
/etc/rc.d/rwho
/etc/rc.d/timed
/etc/rc.d/ugidfw
/etc/rc.d/yppasswdd
/etc/rc.d/utx
/etc/rc.d/LOGIN
/usr/local/etc/rc.d/salt_syndic
/usr/local/etc/rc.d/salt_minion
/usr/local/etc/rc.d/salt_master
/usr/local/etc/rc.d/rsyncd
/usr/local/etc/rc.d/poudriered
/usr/local/etc/rc.d/mdnsresponderposix
/usr/local/etc/rc.d/mdnsd
/usr/local/etc/rc.d/lighttpd
/etc/rc.d/ypxfrd
/etc/rc.d/ypupdated
/etc/rc.d/wpa_supplicant
/etc/rc.d/ubthidhci
/etc/rc.d/syscons
/etc/rc.d/swaplate
/etc/rc.d/sshd
/etc/rc.d/sendmail
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg
/etc/rc.d/securelevel
/etc/rc.d/power_profile
/etc/rc.d/othermta
/etc/rc.d/nfscbd
/etc/rc.d/natd
/etc/rc.d/msgs
/etc/rc.d/moused
/etc/rc.d/mixer
/etc/rc.d/kpasswdd
/etc/rc.d/kfd
/etc/rc.d/kadmind
/etc/rc.d/ipropd_slave
/etc/rc.d/ipropd_master
/etc/rc.d/inetd
/etc/rc.d/hostapd
/etc/rc.d/gptboot
/etc/rc.d/geli2
/etc/rc.d/ftpd
/etc/rc.d/ftp-proxy
/etc/rc.d/dhclient
/etc/rc.d/bsnmpd
/etc/rc.d/bluetooth
/etc/rc.d/bgfsck
/etc/rc.d/autounmountd
/etc/rc.d/automount
/etc/rc.d/automountd
 
Ok. Thanks for the information. What I was looking for is if you were running DNS software on the local machine that the DNS software is started before /etc/rc.d/mountcritremote executes.

In this case, that shouldn't be an issue. However I think I'm a bit lost on one piece since the thread started mentioning both NFS clients and an NFS server. Is this one of the clients? If so, does it use DHCP? An /etc/rc.conf option on an NFS client with DHCP may be to use:
Code: 
ifconfig_<interface>="SYNCDHCP"

The logic is the synchronous DHCP option holds up the boot process until it gets an address and this will make absolutely sure name resolution is available when mounting remote shares.
 
Both the name server VM (nfs client) and the nfs server physical machine use static IP address and /etc/rc.conf does not contain:
Code: 
ifconfig_<interface>="SYNCDHCP"
 
Grazzy said:
Both the name server VM (nfs client) and the nfs server physical machine use static IP address and /etc/rc.conf does not contain:
Code: 
ifconfig_<interface>="SYNCDHCP"
Click to expand...

The static IPs should ensure the name lookup for the NFS server's hostname works. At the moment I am out of solid ideas on what this could be.
 
junovitch@ said:
At the moment I am out of solid ideas on what this could be.
Click to expand...
That's fine. Now that I am able to boot the machine with the workaround, there are other fires to put out and this issue has dropped to a lower priority. Thanks again for the help!
 
You must log in or register to reply here.
Back
Top