HOWTO: Setup diskless booting of Raspberry Pi (running RPi OS) via TFTP & NFSv4 from a FreeBSD ZFS server

Peter Eriksson

Peter Eriksson

Version: 1.0 (2024-03-14)
Author: Peter Eriksson <pen@lysator.liu.se>

Below is a guide on how to setup a Raspberry Pi 4B running Raspberry Pi OS 12 for PXE network boot with diskless NFSv4
root using ZFS for fast cloning of boot environments on a FreeBSD 13 server.

BEWARE:
This requires changing two sysctl kernel settings that enables use of uid & gid numbers over the NFSv4 protocol with sec=sys.
If you already use sec=sys for other clients then this might break stuff. But you don't do that, do you? sec=krb5 is the way to go... :)


Prerequisites / Environment
  1. FreeBSD server with a ZFS data pool mounted on /export ($ZPOOL)
  2. Raspberry Pi
  3. Micro SD Card
  4. A DHCP server that gives out IP adresses to the Raspberrys ($CLIENTIP)
Remember the server IPv4 address ($SERVERIP)


Step 1 - Prepare OS on Raspberry

1. Download the latest Raspberry Pi OS installer from www.raspberrypi.com/software and install the OS on a SD card.

2. Configure some OS customization settings (not 100% needed but is nice to do):
  • General:
    • Set hostname: HOSTNAME.local
    • Set username and password:
      • Username: admin
      • Password: <secret>
  • Services:
    • Enable SSH (use password authentication)

3. Write the OS to the SD card.

4. Insert SD card into Raspberry and connect it to a screen, network & keyboard

5. Boot and login as the admin user on the Raspberry

6. Update the operating system:

apt update ; apt upgrade -y

7. Get/save the Raspberry serial number ($SERIAL):

vcgencmd otp_dump | grep 28: | sed s/.*://g

8. Get/save the MAC ($CLIENTMAC) & IP address ($CLIENTIP):

ip a list

9. In order to successfully be able to NFS mount from a NFSv4 server a hack/hook needs to be installed in the initramfs image since the default "/usr/lib/klibc/bin/nfsmount" binary only supports NFSv2 & NFSv3. However, the normal /usr/sbin/mount.nfs binary is compatible and works so the hooks replaces nfsmount with mount.nfs.

Put the following script into /usr/share/initramfs-tools/hooks as "zz-nfs":

Code: 
#!/bin/sh
[ prereqs = "$1" ] && exit
. /usr/share/initramfs-tools/hook-functions
rm -f ${DESTDIR}/bin/nfsmount
copy_exec /sbin/mount.nfs /bin/nfsmount

Remember to make it executable, and then run the command:

update-initramfs -u -v

10. Configure boot order via "raspi-config":
  • 6. Advanced Options -> A5 Bootloader Version -> E1 Latest
  • 6. Advanced Options -> A4 Boot Order -> B3 Network
11. Reboot once so the EEPROM is reflashed (leave the SD card in the Raspberry)


Step 2 - Prepare some NFS shares on the file server & the TFTP server:

1. Activate the kernel nfs settings:

Add the following to /etc/sysctl.conf:
Code: 
vfs.nfsd.enable_stringtouid = 1
vfs.nfs.enable_uidtostring = 1

And also activate them manually:
Code: 
sysctl vfs.nfsd.enable_stringtouid=1
sysctl vfs.nfs.enable_uidtostring=1

2. Create /export/tftpboot and /export/netboot directories:

Code: 
zfs create -o sharenfs="sec=sys ro -maproot=root" $ZPOOL/tftpboot
zfs create -o sharenfs="sec=sys ro -maproot=root" $ZPOOL/netboot

3. Create template directories for the data from the Raspberry Pi:

Code: 
zfs create -o sharenfs="sec=sys rw -maproot=root $CLIENTIP" $ZPOOL/netboot/template
zfs create -o sharenfs="sec=sys rw -maproot=root $CLIENTIP" $ZPOOL/tftpboot/template

4. Enable the TFTP server:

Code: 
cat >/etc/rc.conf.d/tftp <<EOF
tftpd_enable="YES"
tftpd_flags="-s /export/tftpboot"
EOF
service tftpd start

5. Configure & enable the NFS server:

Code: 
cat >>/etc/rc.conf <<EOF
nfs_server_enable="YES"
nfsv4_server_enable="YES"
EOF

cat >/etc/rc.conf.d/mountd <<EOF
mountd_enable="YES"
EOF

service nfsd start
service mountd start

Step 3 - Copy all the files from the Raspberry to the server:

Code: 
mount -t nfs -o sec=sys,vers=4 $SERVERIP:/netboot/template /mnt
rsync -ax / /mnt
umount /mnt
mount -t nfs -o sec=sys,vers=4 $SERVERIP:/tftpboot/template /mnt
rsync -ax /boot/firmware/ /mnt
umount /mnt


Step 4 - Disable NFS sharing of the template directory and make an initial client clone:

Code: 
zfs set sharenfs=off $ZPOOL/netboot/template
zfs set sharenfs=off $ZPOOL/tftpboot/template


Step 5 - Clean up the template etc/fstab file and make a snap to be used for clones:

Code: 
echo "proc /proc proc defaults 0 0" >/export/netboot/template/etc/fstab


Step 6 - Make template snapshots to be used as a source for clone clients:

Code: 
zfs snap $ZPOOL/netboot/template@default
zfs snap $ZPOOL/tftpboot/template@default


Step 7 - Make a clone of the template for a first Raspberry client:

Code: 
zfs clone -o sharenfs="sec=sys rw -maproot=root $CLIENTIP" $ZPOOL/netboot/template@default $ZPOOL/netboot/$SERIAL
zfs clone -o sharenfs="sec=sys rw -maproot=root $CLIENTIP" $ZPOOL/tftpboot/template@default $ZPOOL/tftpboot/$SERIAL


Step 8 - Modify some files in the client root:
Make up some name for the Raspberry ($NAME).

Code: 
cd /export/netboot/$SERIAL
echo $NAME >etc/hostname
echo $NAME >etc/mailname
sed -i bak -e "s/HOSTNAME/${NAME}/" etc/hosts
echo "$SERVERIP:/tftpboot/$SERIAL /boot/firmware nfs default,vers=4.2 0 0" >>etc/fstab


Step 9 - Set up the TFTP bootcode & host symlink:

Code: 
cd /export/tftpboot
cp ../netboot/template/boot/firmware/bootcode.bin .
chmod a+r bootcode.bin
ln -s ../netboot/$SERIAL/boot/firmware $SERIAL

Step 10 - Point nfsroot to the server in cmdline.txt:

Code: 
cd /export/tftpboot/$SERIAL
echo "console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=$SERVERIP:/netboot/$SERIAL,vers=4.2 rw ip=dhcp elevator=deadline rootwait" >cmdline.txt


Step 11 - Configure the DHCP server:

This varies depending on the DHCP server used but the following settings is required:

  • PXE/BOOTP settings:
    • Boot File: bootcode.bin
    • Next Server: $SERVERIP
  • IPv4 DHCP Options:
    • vendor-encapsulated-options (43): 6:1:3:a:4:0:50:58:45:9:14:0:0:11:52:61:73:70:62:65:72:72:79:20:50:69:20:42:6f:6f:74:ff
    • vendor-class-identifier (60): PXEClient

The vendor-encapsulated-options is required or the Raspberry will not attempt to download
the bootcode.bin file from the TFTP server. It is basically the equivalent of the string "Raspberry Pi Boot".


Step 12 - Remove the SD card from the Raspberry and power cycle it:

Now it should boot from network.
 
You must log in or register to reply here.
Back
Top