Solved Diskless Operation

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

I'm following the guide in 'The Handbook' for setting up diskless operation, but can't figure out what should be copied to 'rootpath'.

31.8.2. Configuring the DHCP Server
Code:
subnet 192.168.0.0 netmask 255.255.255.0 {
   range 192.168.0.2 192.168.0.3 ;
   option subnet-mask 255.255.255.0 ;
   option routers 192.168.0.1 ;
   option broadcast-address 192.168.0.255 ;
   option domain-name-servers 192.168.35.35, 192.168.35.36 ;
   option domain-name "example.com";

   # IP address of TFTP server
   next-server 192.168.0.1 ;

   # path of boot loader obtained via tftp
   filename "FreeBSD/install/boot/pxeboot" ;

   # pxeboot boot loader will try to NFS mount this directory for root FS
   option root-path "192.168.0.1:/b/tftpboot/FreeBSD/install/" ;

}
Can I just extract an ISO image into FreeBSD/install/ or do I need to build a special diskless version of FreeBSD?

It mentions:-
Install the base system into ${NFSROOTDIR}, either by decompressing the official archives or by rebuilding the FreeBSD kernel and userland (refer to Section 23.5, “Updating FreeBSD from Source” for more detailed instructions, but do not forget to add DESTDIR=${NFSROOTDIR} when running the make installkernel and make installworld commands.
I've run
Code:
tar xf FreeBSD-11.1-RELEASE-i386-bootonly.iso  -C /tftpboot/FreeBSD
which extracts the rootfs to /tftpboot/FreeBSD and have set 192.168.1.2:/tftpboot/FreeBSD as the root-path.

When booting, the PXE client executes the PXE boot loader then stops with a
NFS MOUNT RPC error: 72
/
can't load 'kernel'

Type '?' for a list of commands.
Any ideas? How can I tell which file is being sought, ie precise path... Is it /tftpboot/FreeBSD/boot/kernel/kernel ?
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

I have done network diskless boot with GNU/Linux but the principle should be the same.
Normally in the tftpboot directory one places a squash file system (memdisk), which the same as the initial ramdisk when loading the Linux kernel. The diskless system gets its IP from the DHCP and downloads a kernel and the ramdisk via the network. Then it loads the kernel and provides the ramdisk to it.
FreeBSD should be able to do the same thing but I have never done it.
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

OK, the documentation explains what exactly happens in FreeBSD:
When a computer PXE boots, it receives information over DHCP about where to obtain the initial boot loader file. After the host computer receives this information, it downloads the boot loader via TFTP and then executes the boot loader. In FreeBSD, the boot loader file is /boot/pxeboot. After /boot/pxeboot executes, the FreeBSD kernel is loaded and the rest of the FreeBSD bootup sequence proceeds, as described in Chapter 12, The FreeBSD Booting Process.
So the bootloader is downloaded via TFTP and then it kicks off the process.
And also what to install:
Install the base system into ${NFSROOTDIR}, either by decompressing the official archives or by rebuilding the FreeBSD kernel and userland (refer to Section 23.5, “Updating FreeBSD from Source” for more detailed instructions, but do not forget to add DESTDIR=${NFSROOTDIR} when running the make installkernel and make installworld commands.
So what you need to extract there is the base.txz downloaded for example from here: https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

I understand the principle, its just a matter of figuring it out in practice. The problem is probably down to permissions of $NFSROOTDIR and what exactly should be copied there...
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

I have read the docu. I think two different things are mixed there. One is having a network boot in order to start a FreeBSD installation, and the other is just running the system in diskless mode and storing the files on a NFS share.
If you want to do the second, I think you don't need a FreeBSD installation medium. Just expand the base.txz, probably also lib32.txz and src.txz etc. depending on what do you want to install there, but base is a must.
Then you need to adjust these two options in /usr/local/etc/dhcpd.conf:
Code:
   # path of boot loader obtained via tftp
   filename "boot/pxeboot" ;

   # pxeboot boot loader will try to NFS mount this directory for root FS
   option root-path "192.168.0.1:/b/tftpboot/" ;
Also adjust the NFS share. Simply remove "FreeBSD/install" from all the paths and it should work.
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

Oh, something else. In the example there the / is mounted read only. You have to adjust your fstab accordingly if you want to save files to the NFS share. For example, you could create a second share and mount it as "/usr/home" on the diskless system, where the user files can be stored.
Also /var/log is probably necessary to be mounted read/write.
Or if you don't want to bother, you can mount the whole NFS share as read-write. I don't know how exactly the permissions work over NFS.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

I have read the docu. I think two different things are mixed there. One is having a network boot in order to start a FreeBSD installation, and the other is just running the system in diskless mode and storing the files on a NFS share.
I was reading this article which talks setting about up a "jumpstart" server... it uses memory file system as root device, which I was trying to explore, but the instructions are quite old and probably don't apply to FreeBSD 12.0. It keeps referring to 'jumpstart' but I can't find any references to what it is or where to find it.

I had to smirk when I read
  • If you have plenty of RAM, then you can install the base system on a memory file system and load that. The base system takes up some 150 MB and with systems shipping with 1024MB RAM this may be the easiest option.
😏
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

I'm only guessing, but I can't help thinking this 'jumpstart' process was a predecessor mfsBSD.... and I already have that booting via PXE.

As for the normal diskless client, I've made progress and get as far as the menu prompt, but then there is a msg
can't find 'kernel'
can't find 'kernel'
Error while including /boot/menu.rc, in the line:
menu-display
|
can't load 'kernel'

Type '?' for a list of commands.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

Managed to get it working eventually and in hindsight it was fairly straightforward - the hardest part was getting the permissions right on the NFS server - ie. NFSROOT.

Early tests show that an old ThinkPad X41 will boot FreeBSD 11.2 i386 up to a login prompt in about 45 secs and a newer, more powerful X220 in only a few secs quicker.

Looks like I'll be exploring network booting quite a bit over the forthcoming weeks.
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

Yeah, be careful not to read obsolete docus. FreeBSD has everything it needs for booting in this scenario.
I have a threadripper workstation with tons of cores and RAM, and NVMe. The boot still takes about 40 seconds. 45 is perfectly fine in my opinion.
And also, diskless takes time in general. Unless you have a 10 GBit ethernet, a small delay will always be there. But mostly it's not an issue.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

Now, after following the guide on diskless operation I can do a remote boot, but I'd like to install a few applications on the boot server, as well as running sshd().

There is some mention of how to do this in section 12,13 of

31.8.1. Setting Up the PXE Environment.:-

  1. If needed, enable ssh(1) root logins for client machines which are PXE booting by editing${NFSROOTDIR}/etc/ssh/sshd_config and enabling PermitRootLogin. This option is documented in sshd_config(5).
  2. Perform any other needed customizations of the PXE environment in ${NFSROOTDIR}. These customizations could include things like installing packages or editing the password file with vipw(8).
I can't start sshd() because it can't create keys on a read-only filesystem.

I thought I might be able to install pkgs by chroot()ing into ${NFSROOTDIR}, but it looks like pkg()
doesn't work in that environment.
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

you don't need to chroot, pkg can use a directory prefix for installing. The parameter is "--rootdir".
I wrote above about mounting the /, you could do it as read-only or read-write. Take a look for example here: https://lists.freebsd.org/pipermail/freebsd-questions/2010-February/212069.html
They cite a kernel variable vfs.root.mountfrom.options=rw, which you could add to loader.conf.

Then you get your NFS root in read/write mode.

Optionally, you could mount it as read-only and only when you need to change something, you could remount the root as read-write: https://www.cyberciti.biz/faq/howto-freebsd-remount-partition/
Code:
# mount -o rw /
Of course, your NFS share must be exported with write permissions.

And you always have the option to modify the NFS root on the server side itself, just use pkg with directory prefix. But your server needs to be FreeBSD if you want that.
 
OP
OP
B

balanga

Son of Beastie

Reaction score: 106
Messages: 2,514

I now have my $NFSROOTDIR set up with different versions of FreeBSD extracted into corresponding directories and it seems to work well, so just need to set Root-path for the particular version of FreeBSD I want to run on a particular system, identified by MAC address and it boots. I'm really pleased with it. I'll study the links provided to enhance what can be done with this setup.

One thing I was wondering, is, can one version of pxeboot handle all the different versions. I haven't had a problem so far, just wondered whether to expect one... Maybe always using the latest would be a good idea....
 

roccobaroccoSC

Active Member

Reaction score: 36
Messages: 131

That for sure would be a good idea. The latest version should be capable of handling any other version of the kernel.
Is your goal the ability to switch between FreeBSD versions easily? There is a simpler way to do it.
You could use ZFS as file system and then have separate datasets for each FreeBSD version with complete tree under it. You can put your home outside the versions tree and you can switch between versions while keeping your home the same:

zroot/ROOT/12.0
zroot/ROOT/12.0/boot
zroot/ROOT/12.0/bin
....
zroot/ROOT/11.2
zroot/ROOT/11.2/boot
zroot/ROOT/11.2/bin
...
zroot/usr/home

Then you can use the "canmount" and "mountpoint" properties of the datasets to switch between the versions. It's not fool proof, but if you are careful it's very flexible and easy.
If something gets messed up, you can always boot from a rescue disk and fix the mountpoints.

Wrapping the commands in a script "switch_to_12.sh" and "switch_to_11.sh" would probably save you effort and make things easier.
 
Top