Solved Diskless (PXE) boot stuck

For balanga

As suggested by covacat, I use 10.1.1.1 as the tftpd, nfs, and dhcpd server in a /24 subnet. This falls onto the re0 ethernet interface. Unless stated otherwise, all configuration files refer to this 10.1.1.1 host (not the clients).

I have a zfs dataset named zroot/diskless with a mountpoint of /

I have FreeBSD 15.0-RELEASE boot imaged installed there as /diskless/15.0-RELEASE/
Note that the pxeboot file appears as /diskless/15.0-RELEASE/boot/pxeboot

I purposely choose to have tftpd and nfs share an overlapping directory. The pxeboot image might change from one release to the next so the overlap makes life a little is easier to keep the two in sync. Two config changes are required if, for example, I want to switch to 14.3-RELEASE. This also means I can configure isc-dhcpd to dish out different releases to difference MAC addresses.

$ cat /etc/rc.conf
sh:
inetd_enabled="YES"
# isc-dhcpd
dhcpd_enable="YES"
dhcpd_faces="re0"
# nfs
nfs_server_enable="YES"
rcpbind_enable="YES"
mountd_enable="YES"
rpc_lockd_enable="YES"
rcp_statd_enable="YES"
nfs_server_flags="-b 10.1.1.1 -a 10.1.1.0/24"

I have isc-dhcpd limited to the re0 ethernet interface so that it does not touch wlan0. Likewise, I have nfs limited to to same ip and serving only the /24 subnet

$ grep 'tftp' /etc/inetd.conf
sh:
tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd blocksize 1468 -l -s /diskless/15.0-RELEASE
tftp    dgram   udp6    wait    root    /usr/libexec/tftpd      tftpd blocksize 1468 -l -s /diskless/15.0-RELEASE

One of the man pages suggest the above blocksize. I don't think it helps but neither does it hurt.

For nfs, I have the following /etc/exports file:
sh:
/diskless/15.0-RELEASE  -ro     -maproot=0      -alldirs        -network 10.1.1.0       -mask 255.255.255.0

Here are the relevant bits of /usr/local/etc/dhcpd.conf:

sh:
subnet 10.1.1.0 netmask 255.255.255.0 {
}

group diskless {
  next-server 10.1.1.1;
  filename "boot/pxeboot";
  option root-path "10.1.1.1:/diskless/15.0-RELEASE";
  host xxxx {
    hardware ethernet xx.xx.xx.xx.xx.xx;
    fixed-address 10.1.1.11;
  }
  host yyyy {
    hardware ethernet yy.yy.yy.yy.yy.yy;
    fixed-address 10.1.1.12;
  }
}

Some minor details on the above configuration file.

(tftp) next-server points to the tftpd server.
(tftp) filename includes a "boot/" prefix so that the tftp clients can request it properly.
(nfs) option root-path includes the IP address prefix.

If I don't want to boot from nfs, I can add a "tftp:" prefix but then I must provide a vfs.root.mountfrom to mount a root file system after the kernel has loaded.

As the last config file to show in this post, I have this in /diskless/15.0-RELEASE/etc/fstab:
sh:
# Device        Mountpoint      FStype  Options         Dump    Pass#
10.1.1.1:/diskless/15.0-RELEASE /       nfs     ro      0       0
 
I should point out that I observed the client machine fetch the fstab file. I suspect the kernel uses this to discover the root mount - in the absence of a vfs.root.mountfrom parameter passed to the loader.
 
Apparently UFEI boot doesn't use pxeboot but this does not appear to be relevant to my older hardware. I've only seen this mentioned in the handbook.

For UEFI PXE based boot, the actual boot loader file to use is /boot/loader.efi. See the below section Debugging PXE Problems on how to use /boot/loader.efi.
 
unitrunker I had another look at my system and tried to get it serving up PXEBOOT.

My original aim was to test out an alternative PXE server using my DHCP server. dnsmasq in my case. The principle aim was to set up NEXT-SERVER using a different machine, but I haven't managed to do that. However I thought about what you were doing so thought I'd try the same.

After removing the NEXT-SERVER option and setting up PXEBOOT as the bootfile. I was suprised to see the target PC booting up FreeBSD disklessly.

I didn't do any more testing at that point because my server crashed, but will revisit when things settle down.
 
I've tried three different target machines. PXE boot is consistently broken. I'll log a bug on bugzilla. It will take me a while to document the environment and everything tried. I do hope this is a mistake on my part.
Perhaps this is a network issue. Have you checked the network hardware infrastructure, cable, router, etc.?

I did some diskless testing in emulators/virtualbox-ose-72 VMs (no metal machine available at the moment), no matter if BIOS or UEFI, the diskless machine boots into multi-user mode and is full functional.

I followed the 34.10. Diskless Operation with PXE handbook chapter, installed net/dhcpd as DHCP server instead of net/isc-dhcp44-server (but this shouldn't matter), NFSROOTDIR/etc and NFSROOTDIR/var are gziped.

System setup:

VM PXE server: BIOS/UEFI, 14.3-RELEASE, ZFS, zroot/diskless dataset (UEFI pxeboot is loader.efi renamed).

NFSROOTDIR: 15.0-RELEASE, /diskless/15.0-RELEASE directory

/etc/exports
Rich (BB code):
/diskless/15.0-RELEASE    -ro    -alldirs    -maproot=root

/etc/inetd.conf
Code:
tftp    dgram    udp    wait    root    /usr/libexec/tftpd    tftpd -l -s /diskless/15.0-RELEASE

NFSROOTDIR/etc/fstab
Rich (BB code):
192.168.56.10:/diskless/15.0-RELEASE    /    nfs    ro    0    0

NFSROOTDIR/etc/rc.conf
Code:
hostname="pxe-client.home"


Code:
NFSROOTDIR # ls -F conf/base
etc/        etc.cpio.gz     var/        var.cpio.gz

NFSROOTDIR # cat conf/base/etc/md_size conf/base/var/md_size
40960
40960


DHCP server: net/dhcpd

/usr/local/etc/dhcpd.conf, copied from the handbook, removed some options, left only following settings:
Rich (BB code):
subnet 192.168.56.0 netmask 255.255.255.0 {
   range 192.168.56.20 192.168.56.30 ;
   option subnet-mask 255.255.255.0 ;
   option domain-name "PXE-server.home";

   # IP address of TFTP server
   next-server 192.168.56.10 ;

   # path of boot loader obtained via tftp
   filename "/boot/pxeboot" ;

   # pxeboot boot loader will try to NFS mount this directory for root FS
   option root-path "192.168.56.10:/diskless/15.0-RELEASE/" ;

}

/etc/rc.conf
Rich (BB code):
nfs_server_enable="YES"
inetd_enable="YES"
dhcpd_enable="YES"


PXE Client:

Rich (BB code):
root@pxe-client:~ # less /var/log/messages
Jan 10 13:31:08 pxe-client syslogd: kernel boot file is /boot/kernel/kernel
Jan 10 13:31:08 pxe-client kernel: ---<<BOOT>>---
Jan 10 13:31:08 pxe-client kernel: Copyright (c) 1992-2025 The FreeBSD Project.
Jan 10 13:31:08 pxe-client kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jan 10 13:31:08 pxe-client kernel:     The Regents of the University of California. All rights reserved.
Jan 10 13:31:08 pxe-client kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jan 10 13:31:08 pxe-client kernel: FreeBSD 15.0-RELEASE-p1 releng/15.0-n280999-63a5b645b23e GENERIC amd64
Jan 10 13:31:08 pxe-client kernel: FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
Jan 10 13:31:08 pxe-client kernel: VT(vga): text 80x25
...
Jan 10 13:31:08 pxe-client kernel: Trying to mount root from nfs:192.168.56.10:/diskless/15.0-RELEASE [ro]...
...
Jan 10 13:31:08 pxe-client kernel: NFS ROOT: 192.168.56.10:/diskless/15.0-RELEASE/


root@pxe-client:~ # mount
192.168.56.10:/diskless/15.0-RELEASE/ on / (nfs, read-only)
devfs on /dev (devfs)
tmpfs on /etc (tmpfs, local)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
 
Actually, I've just been able boot FreeBSD disklessly using dnsmasq, and not exactly sure what I did to get it set up.

I must experiment a bit to see how it worked, but it did. I even had LXDE coming up.
 
unitrunker Here's my dnsmasq.conf:-

Code:
dhcp-range=192.168.1.20,192.168.1.29,255.255.255.0,24h
dhcp-option=3,192.168.1.1
dhcp-option=6,0.0.0.0
dhcp-option=option:ntp-server,0.0.0.0
dhcp-range=interface:em0,192.168.1.10,192.168.1.15,24h
no-resolv
server=1.1.1.1
server=8.8.8.8
strict-order
domain-needed
bogus-priv
expand-hosts
domain=lan
dhcp-host=00:1d:72:96:4e:0b,192.168.1.250,x6,24h
enable-tftp
tftp-root=/usr/local/tftproot
dhcp-boot=pxeboot

/etc/exports:=

Code:
/ -mapall="root"

There are probably a few settings in dnsmasq that I don't need, but I'm not familiar with the program to know.

It's kinda minimalist, without any regarde to security but it worked for me.
 
Thank you everyone!

Two mistakes. One is I posted my syslogd flags in place of my actual nfs flags:
$ cat /etc/rc.conf
sh:
nfs_server_flags="-b 10.1.1.1 -a 10.1.1.0/24" # <-- lies!
My actual flags were:
sh:
nfs_server_flags="-uh 10.1.1.1" # <-- wrong!

Recall that I did not want NFS binding to wlan0.

I should have used:
sh:
nfs_server_flags="-tuh 10.1.1.1"

pxeboot only speaks UDP to NFS but mountroot prefers TCP so both must be enabled. Inserting that one character fixed the problem.

I did follow T-Daemon's suggestions on creating a pair of cpio.gz files.

T-Daemon There was a fantastic presentation by Michael Dexter ("call for testing") on using NFS on the lo0 interface to serve VMs. All of the VMs shared an over-subscribed ZFS file system making disk space demands more efficient.


I'm working on a surprisingly small script to (a) generate the /diskless/XX.X-RELEASE disk image and (b) generate a checklist of configuration changes similar to freebsd-update IDS.

balanga I've seen different suggestions for the NFS exports file. Some say -maproot-root, others say maproot=0. I've not taken the time to learn the differences. I also noticed the surpising number of DHCP options. DHCP is like a low-rent LDAP.

Thank you everone!
 
Thank you everyone!



balanga I've seen different suggestions for the NFS exports file. Some say -maproot-root, others say maproot=0. I've not taken the time to learn the differences. I also noticed the surpising number of DHCP options. DHCP is like a low-rent LDAP.

Thank you everone!

I am far from being authority on these things, it's just that I managed to make it work somehow.

Looking at my very basic /etc/rc.conf there is no nfs_server_flags setting, so I'm surprised that it works at all.

In fact I just noticed that I have dnsmasq_iface=em0 because I transferred the disk from my old machine and this one does not have an em0, so I don't why anything works.
 
Back
Top