NFS services refusing client & cannot mount

Beeblebrox

Beeblebrox

NFS share is running inside jail. Services started by jailname/etc/rc.conf:
Code: 
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
rpcbind_enable="YES"
mountd_flags="-r"
mountd_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
Booting from diskless client gave errors and mount NFS on linux client gave error (/ shared on exports for testing):
# mount -t nfs 192.168.2.1:/ /mnt
Code: 
mount.nfs: rpc.statd is not running but is required by for remote locking
mount using '-o nolock' or start rpc.statd

EDIT:
[facepalm] Do you suppose having disabled support for NFS Server & Client in the kernel config file could have something to do with it? [/facepalm]
 
Beeblebrox said:
[facepalm] Do you suppose having disabled support for NFS Server & Client in the kernel config file could have something to do with it? [/facepalm]
Click to expand...
I don't think that's the issue, they'll get loaded as modules automatically.

But make sure all your services are correctly bound to the jail's IP address. Also make sure there aren't any services on the host that might interfere.

Code: 
rpcbind_enable="YES"
rpcbind_flags="-h 192.168.100.190"
rpc_lockd_enable="YES"
rpc_lockd_flags="-h 192.168.100.190"
rpc_statd_enable="YES"
rpc_statd_flags="-h 192.168.100.190"
mountd_enable="YES"
mountd_flags="-h 192.168.100.190"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4 -h 192.168.100.190"
 
The start of nfs gives errors to dmesg:
Code: 
sysctl: vfs.nfsd.nfs_privport: Operation not permitted
NFSv4 is disabled
sysctl: vfs.nfsd.server_max_nfsvers: Operation not permitted
Starting nfsd.
nfsv4(4)() message corrected by adding to jail/etc/rc.conf:
Code: 
nfsv4_server_enable="YES"
Adding this in host /etc/sysctl.conf did not fix the problem:
Code: 
vfs.nfsd.nfs_privport=1
vfs.nfsd.server_max_nfsvers=4
pxe client shows:
Code: 
NFS MOUNT RPC error: 72
can't load kernel
Host Kernel messages - these no longer showed up when I enabled NFSv4:
Code: 
mountd: can't delete exports for /: operation not permitted
mountd: bindresvport_sa: Address already in use
The NIC is only used for jail processes and no servers running on host

EDIT: Seems I'm not the first with this problem: http://forums.freebsd.org/archive/index.php/t-13835.html
 
From a post in May 2010 - I hope this is not the case anymore?
Code: 
You cannot mount NFS shares inside a jail because of rpcbind.
The best solution I've found is to mount the NFS share on the jail host, 
and create a nullfs mount of that mountpoint to the jail.

EDIT: TESTING: Disabled all NFS services in jail but left dhcpd and inetd alive. Started the jail and all nfs from the host side. Start the diskless client, and boots into the BTX loader!
What should be the setting for these sysctl.conf variables in jailed nfs?
Code: 
WARNING: sysctl vfs.nfsd.nfs_privport does not exist.
WARNING: sysctl vfs.nfsd.server_max_nfsvers does not exist.
The jail's messages log has this:
Code: 
rpcbind: cannot bind 192.168.2.1 on udp: Address already in use
rpcbind: cannot bind 192.168.2.1 on tcp: Address already in use
 
Some progress - I think. I disabled all services in the jail's etc/rc.conf and started the jail. Then I jexec'ed into the jail and started the services manually and one-by-one to track the dmesg output. From what I can tell, starting dhcp before nfs services causes nfs to not able to bind to the address. Here's what happened (IPv6 is disabled on system):
Code: 
# service rpcbind onestart -h 192.168.2.1
rpcbind: cannot get information for udp6
rpcbind: cannot get information for tcp6

# service lockd onestart -h 192.168.2.1
starting

# service mountd onestart -h 192.168.2.1
mountd[2189]: Can't delete exports for V4:
mountd[2189]: can't delete exports for /: Operation not permitted 
mountd[2189]: can't change attributes for /
mountd[2189]: bad exports list line /
* The exports file is just one entry: / (export root to everyone, no conditions).
* Please note that in this start-up sequence, rpcbind does not complain about "cannot bind 192.168.2.1 on udp: Address already in use".
* Continuing, sysctl messages:
Code: 
# service nfsd onestart -h 192.168.2.1
sysctl: vfs.nfsd.nfs_privport: Operation not permitted
NFSv4 is disabled
sysctl: vfs.nfsd.server_max_nfsvers: Operation not permitted
nfsd[2238]: Can't read stable storage file

# service nfsclient onestart -h 192.168.2.1
NFS access cache time=60
sysctl: vfs.nfs.access_cache_timeout: Operation not permitted
/etc/rc.d/nfsclient: WARNING: failed to set access cache timeout

# service nfscbd onestart -h 192.168.2.1
nfscbd:[2379]: can't Add UDP socket
nfssvc: Operation not permitted
nfscbd: Can't get fully qualified host name
Code: 
# service isc-dhcpd onestart
starting

# service inetd onestart
inetd[2411]: madvise() failed: Operation not permitted

# service ntpd onestart
ntpd[2417]: ntpd 4.2.4p5-a (1)
ntpd[2418]: bind() fd 21, family AF_INET, port 123, addr 192.168.2.1,
in_classd=0 flags=0x19 fails: Address already in use
ntpd[2418]: unable to create socket on fxp0 (1) for 192.168.2.1#123
# service ntpd onestop

EDIT: we have an improvement. Previously the hanged client (? for list of commands) at btx stage was un-responsive. now responds to ls with:
Code: 
open '/' filed: RPC struct is bad
Host's reason why client boot did not go forward:
Code: 
mountd[2190]: mount request denied from 192.168.2.7 for /
 
Beeblebrox said:
From a post in May 2010 - I hope this is not the case anymore?
Code: 
You cannot mount NFS shares inside a jail because of rpcbind.
The best solution I've found is to mount the NFS share on the jail host, 
and create a nullfs mount of that mountpoint to the jail.
Click to expand...

That shouldn't be the case anymore. You can bind rpcbind to a specific IP address now. It used to grab all IP addresses.
 
CORRECTION: I ran all three services (dhcp-inetd-nfsd) from outside the jail and started the diskless client. The kernel loads and gets to the btx loader screen but stops at mount root. "?" command - which is supposed to show all devices shows NOTHING - no geom managed disk devices.

PROGRESS: I found several mistakes and corrected. The boot/loader.conf being exported to the pxe-client should have included
Code: 
nfscl_load=yes
vfs.root.mountfrom="nfs:192.168.2.1/usr/jails/pxeboot"
Without nfs client module (nfscl.ko) you will get message
Code: 
mount root failed with error 2: unknown file system
These adjustments are not the end of the problems as yet because I now get an "error 19" mount fail message, which turns out to be a ZFS issue. Looks like I need to re-locate the exported root to its own filesystem under zfs
# zfs set sharenfs=on tank/jailedpxeboot
EDIT: That's a dead-end because zfs' sharenfs is just another method of doing what etc/exports does.
If set to off, the file system is managed by using the legacy share and unshare commands
Click to expand...
 
When booting the pxe-client, the process gets to the btx loader menu. There I select "2: loader prompt". At the prompt, lsdev (list devices) shows
Code: 
pxeboot - 192.168.2.1:/boot/pxeboot
but no nfs devices. Can someone confirm what devices are visible to diskless clients at that boot stage?


EDIT:
I don't think that's the issue, they'll get loaded as modules automatically.
Click to expand...
Not all of them, apparently: This MUST be in the kernel. Without it MOUNT NFS fails.
Code: 
options 	NFS_ROOT		# NFS usable as /, requires NFSCL
NFSCL does not need to be in kernel, it can be loaded as module. Careful: FreeBSD-9 has NFS-v4 as opposed to previous FreeBSD's which do not have v4, and therefore the kernel syntax (NFSCL vs NFSCLIENT) is different. If you build a FreeBSD-9 kernel with NFSCLIENT you will get a version mismatch.
Related other debug thread here.

EDIT2:
I have received some information from the FreeBSD mailing list and apparently exporting NFS from jailed environment is not possible. For those who have "managed" (by heavy tweaking of sysctl.conf) to export the NFS probably have these concerns: 1) Security may have been compromised on their own jails as a result of tweaks and 2) Even if you manage to export the NFS share under such strained boundary conditions, it may cause problems in some of the aps you would like to use (eg: tinderbox) finally, 3) If you try to use net/unfs3 and succeed to export NFS, this will not have a very fast transport rate and will have many speed limitations.
Overall, none of which present a solution to what I am trying to put together.
 
You must log in or register to reply here.
Back
Top