Can't NFS mount exported root

pathiaki · Sep 29, 2013

Hi,

I believe I have read just about everything on booting a diskless client in the Handbook, articles, HOWTOs, searching the forums, etc. I'm on FreeBSD 9.2-RC4. My tftpd line in /etc/inetd.conf

Code:

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /diskless/FreeBSD/boot

My entry for the host and everything else in /usr/local/etc/dhcpd.conf:

Code:

ddns-update-style none;

subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.64 192.168.0.80;
  default-lease-time 3600;
  max-lease-time 86400;

  option domain-name-servers ns.<domain>.<TLD>;
  option routers 192.168.0.1;

}

host debsmachine {
  next-server 192.168.0.2;
  filename "pxeboot";
  option root-path "192.168.0.2:/diskless/FreeBSD/";
  option subnet-mask 255.255.255.0;

  hardware ethernet 78:ac:c0:a7:66:a0;
  fixed-address 192.168.0.77;
}

My server is running NFSv4. The relevant entries in the /etc/rc.conf file are:

Code:

mountd_enable="YES" 
mountd_flags="-r" 
nfs_server_enable="YES" 
nfs_server_flags="-u -t -n 4 -h 192.168.0.2"
nfsuserd_enable="YES" 
nfsv4_server_enable="YES" 
nfscbd_enable="YES"
rpcbind_enable="YES" 
rpcbind_flags="-h 192.168.0.2"

Since I'm using ZFS, I use:

zfs set sharenfs="alldirs,ro,maproot=0" <pool>/diskless/FreeBSD
zfs set sharenfs="alldirs,ro,maproot=0" <pool>/usr

Which creates the following entries in /etc/zfs/exports:

Code:

/diskless	-ro -maproot=0 -alldirs 
/usr	-ro -maproot=0 -alldirs 
/usr/home	-ro -maproot=0 -alldirs 
/usr/ports	-ro -maproot=0 -alldirs 
/usr/src	-ro -maproot=0 -alldirs

In the file, /diskless/FreeBSD/etc/fstab, I have the following:

Code:

192.168.0.2:/usr                        /usr            nfs     ro      0       0
192.168.0.2:/diskless/FreeBSD           /               nfs     ro      0       0
proc                                    /proc           procfs  rw      0       0

In the file, /diskless/FreeBSD/etc/rc.conf, I have the following:

Code:

hostid_enable="NO"
nfs_client_enable="YES"
ntpupdate_enable="YES" 
rpcbind_enable="YES" 
rpcbind_lockd_enable="YES"
rpcbind_statd_enable="YES"
sendmail_enable="NONE"
sshd_enable="YES" 
usbd_enable="YES"

I have also swapped out the NFSv3 stuff to just have the nfsuserd, nfscbd, and just rpcbind so the client is all NFSv4. I believe that's everything I can give right now.

Here's the fun part:

I get my PXEboot

I get my kernel

Once the kernel is loaded, it hangs.

Code:

Trying to mount root from nfs:192.168.0.2:/diskless/FreeBSD[ro]
mountroot: waiting for device 192.168.0.2:/diskless/FreeBSD
Mounting from nfs:192.168.0.2:/diskless/FreeBSD failed with error 19.
Trying to mount root from from nfs: []
NFS ROOT: 192.168.0.2:/diskless/FreeBSD/

I can mount the /diskless/FreeBSD partition on both an NFSv3 and NFSv4 client.

Can anyone see anything that I've done wrong? I have checked in so many locations that I'm ready to scream. I have to figure out how to paint over the bloody spot on the wall where I've been banging my head. :\

Beeblebrox · Sep 30, 2013

From my past experience, nfs_client as a module on the diskless side does not work. So nfs_client_enable is your problem. It needs to be directly in the kernel. Re-compile the kernel you are using for the diskless client with these two additions:

Code:

options    NFSCL    # Network Filesystem Client
options    NFS_ROOT    # NFS usable as /, requires NFSCLIENT

Then install to your root-exported folder:
# make installkernel DESTDIR=/diskless/FreeBSD

EDIT: You could also try something like

Code:

nfs_root_enable="YES"

I don't know if it will work or if that's the correct syntax, as I chose the re-compile method. Also, your kernel module options are in the wrong location (etc/rc.conf) for diskless setup. They should be placed in Freebsd/boot/loader.conf. How is rc.conf going to be read by the system when root has not been mounted yet? At this stage all that you have done is booted the kernel, but you are instructing the kernel modules necessary to mount the root folder, from a file placed inside that root folder.

Beeblebrox · Sep 30, 2013

You might want to try some of these settings in your /diskless/FreeBSD/Boot/loader.conf (from my own setup). I'm also posting the hashed-out entries so you can see options that are not necessary:

Code:

autoboot_delay="-1"
verbose_loading="YES"
boot.nfsroot.server="192.168.2.1"
boot.nfsroot.path="/data/amd64"
vfs.root.mountfrom="nfs:192.168.2.1:/data/amd64"
#vfs.root.mountfrom="nfs"
#boot.nfsroot.options="nolockd"
#vfs.root.mountfrom.options="ro"

pathiaki · Sep 30, 2013

@Beeblebrox,

Thank you for a couple of things that were obvious... (yes, I get a 'duh' for not putting those things in the loader.conf). However, it's still a no-go. Same behavior. I already had the options in the kernel for the client. I have decided to go back to GENERIC and install that. Still the same behavior.

My /diskless/FreeBSD/boot/boot.loader now looks like:

Code:

autoboot_delay="3"
boot.nfsroot.server="192.168.0.2"
boot.nfsroot.path="/diskless/FreeBSD"
hostid_enable="NO"
nfs_root_enable="YES"
ntpupdate_enable="YES"
nullfs_load="YES" 
rpcbind_enable="YES"
tmpfs_load="YES" 
usbd_enable="YES"
verbose_loading="YES"
vfs.root.mountfrom="nfs:192.168.0.2:/diskless/FreeBSD"

However, as I stated, same thing. Do you see anything amiss elsewhere? How does the dhcpd.conf file look?

Beeblebrox · Sep 30, 2013

My /diskless/FreeBSD/boot/boot.loader now looks like:

Is that a typo? You of course meant /diskless/FreeBSD/boot/loader.conf

Your DHCP looks fine - Since the process gets to the point of

Code:

Trying to mount root from from nfs: []

this means the kernel has loaded and DHCP's job is finished. Although if you want, you can include this for PC name being passed with IP:

Code:

use-host-decl-names on

Lets check several other potentially "obvious" problems if you like:
* What does showmount -e form host give you? All normal?
* If you connect a laptop or other PC to your 192.168.0.2 network, does that machine get assigned an IP?
* Is the driver for "debsmachine" nic card & model in the kernel (for example if_re)? If not that driver should be specified in loader.conf

EDIT: Also try with a disabled rpcbind and see if root gets mounted.

pathiaki · Sep 30, 2013

Yes, that was a typo. :r

showmount -e

Code:

Exports list on localhost:
/usr/src                           Everyone
/usr/ports                         Everyone
/usr/home                          Everyone
/usr                               Everyone
/diskless                          Everyone

showmount -a

Code:

All mount points on localhost:
debsmachine.<domain>:/diskless/FreeBSD

Removing rpcbind from /diskless/FreeBSD/boot/loader.conf doesn't change a thing.

However, I'm using the GENERIC 9.2-RC4 kernel now so the kitchen sink is in there. Here's something that I didn't notice before or is recent.

Code:

re0: link state changed to DOWN

What the heck? Why would it down the interface?

This happens right after the

Code:

Trying to mount root from nfs: []

Beeblebrox · Oct 1, 2013

What the heck? Why would it down the interface?

That's normal. If you do a verbose boot of your server, you'll see that the network service is first brought down, and then brought back up with final network settings.

I think your problem is this export line from showmount:

Code:

/diskless         Everyone

As I recall, the folder which is to be root should have its direct export line. Meaning, you must have an export which shows:

Code:

/diskless/FreeBSD         <export parameters>

I also just remembered that you must modify the /etc/hosts.allow. You did allow access from your private network, right? If not, just for now place this at the top (after the explanatory hash-marks end):

Code:

ALL : 192.168.0.0/255.255.255.0 : allow

Finally, In order to eliminate probable NFS-Mount related problems, do <ctrl>+<alt>+<F1> and drop to tty0 when the client starts to boot. If there are any NFS related errors, you should see them there. You can also look through the NFS logfile in /var/log (I forget where it gets logged).

If none of these work, we'll move on to how you have /var mounted on client side (must be writable).

pathiaki · Oct 2, 2013

G'day @Beeblebrox,

Yes, I forgot about the down and up of the interface when booting. I guess I'm getting desperate to find anything at this point. Well, our troubleshooting has changed something. I'm using ZFS on the machine as a whole. Before, when I had /diskless/FreeBSD as a mount, it would see nothing and refuse to even attempt to mount and fail. So, I had changed it to just /diskless and exported with the 'alldirs' flag. Now, however, everything seems ok. I've created <pool>/diskless/FreeBSD and exported it with

Code:

zfs sharenfs="alldirs,ro,maproot=0" <pool>/diskless/FreeBSD

This now works, but it still has the same problem with hanging with error 19 in the same place. x(

As for the /etc/hosts.allow file, I removed the comment at the beginning of the file that allows everything. Still no joy.

Also, as for my /etc and /var directories, I thought rc.diskless took care of creating the memory file systems from whatever was in /diskless/FreeBSD/conf/base/ where I have two files etc.cpio.gz and var.cpio.gz. I created them due to Handbook section 31.8.2 with these commands executed from /diskless/FreeBSD:

chroot /diskless/FreeBSD
mkdir conf/base
tar cvf conf/base/etc.cpio.gz --format cpio --gzip etc
tar cvf conf/base/var.cpio.gz --format cpio --gzip var

Also, no logs show anything wrong on the server.

/var/log/xferlog shows the pxeboot being tftp'd over properly. /var/log/debug.log has nothing relevant. /var/log/messages has something relevant to the declarations with static and dynamic overlap for the 192.168.0.77 address in the /usr/local/etc/dhcpd.conf. I moved the dynamic range to extend only to 71. /var/log/auth.log has nothing relevant.

So, while in /var/log: grep -i nfs *. The only thing that came back was from the mount.* files giving descriptions of the mounted filesystems and the fact they were ZFS and had nfsv4acls. Also, is there anything necessary in /diskless/FreeBSD/etc/fstab to be mounting the /etc and /var? I have nothing at this point.

Is any of this relevant:

My server is using NFSv4 to serve.
My filesystems are all ZFS.
I'm using a the GENERIC kernel from 9.2-RC4 in its default form.

And now for the part when I said, "why haven't I been dumping the line?"

So, on the server, the interface I'm on is em0. We already know the client's MAC address and the IP it's getting, so tcpdump -vvv -n -e -ttt -i em0 | egrep -i '78:ac:c0:a7:66:a0|192.168.0.77':

Code:

tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:00.372426 78:ac:c0:a7:66:a0 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 0, offset 0, flags [none], proto UDP (17), length 576)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 78:ac:c0:a7:66:a0, length 548, xid 0xc1a766a0, secs 4, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
	  Your-IP 192.168.0.77
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
00:00:01.372246 78:ac:c0:a7:66:a0 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 1, offset 0, flags [none], proto UDP (17), length 576)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 78:ac:c0:a7:66:a0, length 548, xid 0xc1a766a0, secs 4, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
	    Requested-IP Option 50, length 4: 192.168.0.77
	  Your-IP 192.168.0.77
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
00:00:00.002359 78:ac:c0:a7:66:a0 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.0.2 tell 192.168.0.77, length 46
00:00:00.000009 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 192.168.0.2 is-at 00:25:90:8a:fd:6e, length 28
00:00:00.000083 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 20, id 2, offset 0, flags [none], proto UDP (17), length 52)
    192.168.0.77.2070 > 192.168.0.2.69: [udp sum ok]  24 RRQ "pxeboot" octet tsize 0 
00:00:00.002597 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 57: (tos 0x0, ttl 64, id 47462, offset 0, flags [none], proto UDP (17), length 43)

Yes, it gets PXEboot but we knew that.

Code:

*snip*

00:00:00.026101 78:ac:c0:a7:66:a0 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 165, offset 0, flags [none], proto UDP (17), length 576)
    192.168.0.77.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 78:ac:c0:a7:66:a0, length 548, xid 0xa066a7c1, Flags [none] (0x0000)
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
00:00:00.000112 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 360: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 346)
    192.168.0.2.67 > 192.168.0.77.68: [udp sum ok] BOOTP/DHCP, Reply, length 318, xid 0xa066a7c1, Flags [none] (0x0000)
	  Your-IP 192.168.0.77
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
00:00:00.000271 78:ac:c0:a7:66:a0 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 590: (tos 0x0, ttl 20, id 166, offset 0, flags [none], proto UDP (17), length 576)
    192.168.0.77.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 78:ac:c0:a7:66:a0, length 548, xid 0xa066a7c1, Flags [none] (0x0000)
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
	    Requested-IP Option 50, length 4: 192.168.0.77
00:00:00.000050 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 360: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 346)
    192.168.0.2.67 > 192.168.0.77.68: [udp sum ok] BOOTP/DHCP, Reply, length 318, xid 0xa066a7c1, Flags [none] (0x0000)
	  Your-IP 192.168.0.77
	  Client-Ethernet-Address 78:ac:c0:a7:66:a0
00:00:00.006086 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 20, id 167, offset 0, flags [none], proto UDP (17), length 104)
    192.168.0.2.111 > 192.168.0.77.1023: [udp sum ok] UDP, length 28
00:00:00.000139 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 142: (tos 0x0, ttl 20, id 170, offset 0, flags [none], proto UDP (17), length 128)
    192.168.0.77.4 > 192.168.0.2.2049: 100 lookup fh Unknown/DCB83973DE076F150A00040000000000DC210C00000000000000000000000004 "boot"
00:00:00.000047 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 278: (tos 0x0, ttl 64, id 47667, offset 0, flags [none], proto UDP (17), length 264)
    192.168.0.2.2049 > 192.168.0.77.4: reply ok 236 lookup fh Unknown/DCB83973DE076F150A003E0000000000E1210C00000000000000000000000001 DIR 755 ids 0/0 sz 41 nlink 7 rdev 33/786652 fsid 7339b8dc fileid 3e a/m/ctime 1380670070.405780166 1380545986.579779000 1380667510.28797865 post dattr: DIR 755 ids 0/0 sz 32 nlink 19 rdev 154/-1288175392 fsid 7339b8dc fileid 4 a/m/ctime 1380669717.55780813 1380668623.935788084 1380668623.935788084

Yes, we found boot.

Code:

    192.168.0.77.14 > 192.168.0.2.2049: 108 lookup fh Unknown/DCB83973DE076F150A003E0000000000E1210C00000000000000000000000009 "loader.rc"
00:00:00.000024 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 278: (tos 0x0, ttl 64, id 47683, offset 0, flags [none], proto UDP (17), length 264)
    192.168.0.2.2049 > 192.168.0.77.14: reply ok 236 lookup fh Unknown/DCB83973DE076F150A00810500000000E6210C00000000000000000000000001 REG 444 ids 0/0 sz 390 nlink 1 rdev 33/786657 fsid 7339b8dc fileid 581 a/m/ctime 1380671532.635793556 1380469997.000000 1380667509.350884540 post dattr: DIR 755 ids 0/0 sz 41 nlink 7 rdev 154/-1288175392 fsid 7339b8dc fileid 3e a/m/ctime 1380670070.405780166 1380545986.579779000 1380667510.28797865
00:00:00.000179 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 146: (tos 0x0, ttl 20, id 181, offset 0, flags [none], proto UDP (17), length 132)

Yes, loader.rc.

Code:

Unknown/DCB83973DE076F150A003E0000000000E1210C0000000000000000000000000A "loader.4th"
00:00:00.000024 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 278: (tos 0x0, ttl 64, id 47696, offset 0, flags [none], proto UDP (17), length 264)
    192.168.0.2.2049 > 192.168.0.77.23: reply ok 236 lookup fh Unknown/DCB83973DE076F150A00910500000000E6210C00000000000000000000000001 REG 444 ids 0/0 sz 5881 nlink 1 rdev 33/786657 fsid 7339b8dc fileid 591 a/m/ctime 1380671532.644779918 1380400070.000000 1380667510.28797865 post dattr: DIR 755 ids 0/0 sz 41 nlink 7 rdev 154/-1288175392 fsid 7339b8dc fileid 3e a/m/ctime 1380670070.405780166 1380545986.579779000 1380667510.28797865

Yes, loader.4th.

It tries to retrieve boot, loader.4th and miscellaneous files. It tries to load all the non-existent gzip'd versions of the files, (failing due to them not existing). It finds loader.conf and grabs it. It looks for a file called loader.conf.local? *shrug* Not there, issues a warning. It goes through the kernel with lots of output there. So, where's the problem? I see a huge amount of traffic going across port 2049 to/from the client/server of this form:

Code:

00:00:00.000234 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 146: (tos 0x0, ttl 20, id 20068, offset 0, flags [none], proto UDP (17), length 132)
    192.168.0.77.19898 > 192.168.0.2.2049: 104 read fh Unknown/DCB83973DE076F150A00850500000000E6210C00000000000000000000000000 512 bytes @ 2560
00:00:00.000023 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 226: (tos 0x0, ttl 64, id 3038, offset 0, flags [none], proto UDP (17), length 212)
    192.168.0.2.2049 > 192.168.0.77.19898: reply ok 184 read REG 444 ids 0/0 sz 2615 nlink 1 rdev 147/272498928 fsid 7339b8dc fileid 585 a/m/ctime 1380681958.761779567 1380400070.000000 1380667509.389779656 55 bytes EOF
00:00:00.000169 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 146: (tos 0x0, ttl 20, id 20069, offset 0, flags [none], proto UDP (17), length 132)
    192.168.0.77.19899 > 192.168.0.2.2049: 104 read fh Unknown/DCB83973DE076F150A00850500000000E6210C00000000000000000000000000 457 bytes @ 2615
00:00:00.000016 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 170: (tos 0x0, ttl 64, id 3039, offset 0, flags [none], proto UDP (17), length 156)
    192.168.0.2.2049 > 192.168.0.77.19899: reply ok 128 read REG 444 ids 0/0 sz 2615 nlink 1 rdev 147/272498928 fsid 7339b8dc fileid 585 a/m/ctime 1380681958.761779567 1380400070.000000 1380667509.389779656 0 bytes EOF
00:00:00.000163 78:ac:c0:a7:66:a0 > 00:25:90:8a:fd:6e, ethertype IPv4 (0x0800), length 146: (tos 0x0, ttl 20, id 20070, offset 0, flags [none], proto UDP (17), length 132)
    192.168.0.77.19900 > 192.168.0.2.2049: 104 read fh Unknown/DCB83973DE076F150A00850500000000E6210C00000000000000000000000000 512 bytes @ 2615
00:00:00.000015 00:25:90:8a:fd:6e > 78:ac:c0:a7:66:a0, ethertype IPv4 (0x0800), length 170: (tos 0x0, ttl 64, id 3040, offset 0, flags [none], proto UDP (17), length 156)
    192.168.0.2.2049 > 192.168.0.77.19900: reply ok 128 read REG 444 ids 0/0 sz 2615 nlink 1 rdev 147/272498928 fsid 7339b8dc fileid 585 a/m/ctime 1380681958.761779567 1380400070.000000 1380667509.389779656 0 bytes EOF

and then it just stops. This seems to coincide with the hang we are seeing.

Sorry, I'm tired while writing this. However, it seems to have loaded PXEboot, all the loader files and the kernel. Also, if I believe the last file it finds before hanging is shortcuts.4th file

Again, thanks for helping me get through this.

P.

Beeblebrox · Oct 2, 2013

Well, our troubleshooting has changed something. I'm using ZFS on the machine as a whole. Before, when I had /diskless/FreeBSD as a mount, it would see nothing and refuse to even attempt to mount and fail. So, I had changed it to just /diskless and exported with the 'alldirs' flag.

I did not understand this - do you mean that previously you were unable to nfs_mount <pool>/diskless/FreeBSD from a normal host (such as a laptop)?

For /etc and /var directories, the gzio file creation is what I meant by "being writeable". I did not want to confuse the issue by bringing them up too soon. When you switch back to a custom kernel, rememder to include or load the module for tmpfs, as the gzio files depend on it.

I have a very similar setup to yours; known diskless clients assigned static IP and guest clients assigned dynamic IP form pool range. I allow guests to boot FreeBSD over the network in order to clamscan their systems for viruses. The only small differences are, my exported root, DHCP and inetd run from inside a jail named pxe located on exported ZFS. I don't export my host's /usr, since the jail has its own; I just export host /usr/local as read-only. Currently my system is working fine, and terminals are able to use graphical environments.

Since we have eliminated most of the obvious, I have to admit that I'm shooting in the dark this point forward:
* Disable your firewall (you probably already did)
* Try another client (lika a laptop)
* My rpcbind, mountd, nfsd flags:

Code:

rpcbind_flags="-h 192.168.2.1"
mountd_flags="-r -n -p 59 -l -h 192.168.2.1"
nfsd_flags="-u -t -n 4 -h 192.168.2.1"

I don't use the below options for rpcbind. If I recall correctly, lockd had given me some problems. Disable for testing.

Code:

rpcbind_lockd_enable="YES"
rpcbind_statd_enable="YES"

* Try starting the above services manually and make sure all three services are running, in below order:
# service rpcbind onestart
# service mountd onestart
# service nfsd onestart
The reason I suggest this is because I just started mine (I have a script for it) and noticed that I have:

Code:

NFSv4 is disabled           Starting nfsd.

This is normal, because from nfsv4(4)(),

To set up the NFS server that supports NFSv4, you will need to either set the variables in rc.conf(5) as follows:
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"

or start mountd(8) and nfsd(8) without the ``-o'' option, which would force use of the old server. The nfsuserd(8) daemon must also be running.

You will also need to add at least one ``V4:'' line to the exports(5) file for NFSv4 to work.

I don't use the "V4:" syntax, and neither do you from what I saw in your list of exports. So you are not using NFSv4, which is fine because I had tried and found it problematic for the diskless client problem. Since your exports do not appear to be V4, it will cause conflicts with lockd.

EDIT: I should also share my rc.conf contents for the diskless clients:

Code:

root_rw_mount="NO"
ntpdate_enable="YES"
ntpdate_hosts="192.168.2.1"
nfs_client_enable="YES"
kld_list="geom_part_mbr geom_part_ebr msdosfs ext2fs umass uhci ohci ehci fuse"

I discovered a very neat trick for etc.gzio. Place your gzio files in conf/base, but also create conf/default/etc. Place "generic" conf files for clients like rc.conf, fstab , etc in that folder. That way, you don't have to go through gzio file creation each time yo decide to make a minor change in those files. The script for diskless client reads base first, then overrides contents of base with whatever is in default. Finally, you can also place folders named by IP (conf/192.168.0.77/etc), and provide client-level customisation as such folder will be read the very last.

pathiaki · Oct 3, 2013

A little progress....

@Beeblebrox,

Since the change to be NFSv3 on both sides, some things have changed. Most notably, there's a definitive statement from the mount daemon in /var/log/messages:

Code:

Oct  3 02:05:34 atlantisservices mountd[36866]: mount request succeeded from 192.168.0.77 for /diskless/FreeBSD

This was not showing up previously. I grep'd the earlier files. Sadly, it shows up during the loader and not after the machine has loaded the kernel. I still get the same issues as before the NFS error 19 and the like. So, I had to do the 'hit it with the hammer' method. I moved the /diskless/FreeBSD/etc/fstab and the /diskless/FreeBSD/conf/etc/fstab out of the way. No difference. Despite your incredible patience and help, I'm finding this tiring. I'm going to try something a little different. I'm going to start over with your configuration.

To this end, I'm going to post all of my relevant settings in all my files in my next post, with your directory structure to see if it works. If it doesn't, then something is seriously amiss. How could the server possibly see the client, mountd responds to the mount request by the client, it mounts the partition and reads the kernel, yet, can't mount that same partition again as root or any other exported partition?

I need to address this from start to finish and see where it goes. There's too many out of date documents out there for various versions of tftp, NFS, FreeBSD versions, HOWTOs and other things that don't seem to be "simple". Also, the fact that there doesn't seem to be any real "debug" switches to turn on that say "Hey, dummy, this is wrong." :e

So, stubborn person that I am, tomorrow is another day. :\

Thank you, again.

P.

pathiaki · Oct 15, 2013

Hi all!

Well, I'm still stuck. I was diverted to some other things for a while, but, finally, got back to this issue.

I figured it would be a good idea to turn on debugging in /etc/rc.initdiskless to see where I'm getting killed after the root gets mounted.

It doesn't execute.

So, even though I'm getting a success message in /var/log/messages from any client (my desktop, the diskless machine, etc), I need to know why that file is not being executed and go backward from there to find out what the issue is.

Ideas?

Thank you,

P.

Beeblebrox · Oct 17, 2013

Hi,

/etc/rc.initdiskless .... doesn't execute.

This to me means an error inside your "diskless world". Try a completely new build:
# svn up /usr/src
# rm -rf /usr/obj/src
# make buildworld -C /usr/src
# make buildkernel -C /usr/src
# make installkernel DESTDIR=/diskless/FreeBSD -C /usr/src
# make installworld DESTDIR=/diskless/FreeBSD -C /usr/src
# make distribution DESTDIR=/diskless/FreeBSD -C /usr/src

P.S. I know you have experience with these commands, just posting for other readers. Just remember to run make distribution.