FreeBSD and automated PXE Setup

fr33bsd · Aug 15, 2021

Hello people,

I am wondering if there is any new tutorial for automated installation of FreeBSD 13 (or 12.2.) using PXE.

Regular installation tutorials exist, like this older one (in german) and this old one.

This (automated ?) guide goes 404

Searching for "pxe" in this forum ends up in "No results found." (the search link might not work)

Is there any config file for the bsdinstaller to automate the setup procedure - or stuff like that? Or do I have to write the full installation procedure in a bash by myself?

I read something about install.cfg in section 6.1 of the old tutorial. But reaching sysinstall(8) ended up in a dead end:

Sorry, no data found for `sysinstall(8)'.

I found this, but this provides information about pxe and diskless setup and informations about automated installation are missing.

Also in this context incomplete is this guide.

Maybe useful is this mfsbsd. I don't know. But there is another dead end thread.

This how-to is good, but what I am missing here is the part of the automated installation.

Another incomplete reference...

Did anyone of you experience automated rollouts of FreeBSD using PXE?

fr33bsd · Aug 16, 2021

Ah, coming closer: https://wiki.freebsd.org/BSDInstall

and closer ... https://www.freebsd.org/cgi/man.cgi?query=bsdinstall&sektion=&manpath=freebsd-release-ports ...
and closer ... https://forums.freebsd.org/threads/bsdinstall-scripting-guide-tutorial.79105/

But this is old and has to be verified:

Installation scripts are located at /usr/src/usr.sbin/bsdinstall/scripts/ . First file to run is ‘auto’, which then calls other files. Almost all of them are shell scripts.

fundon · Sep 6, 2021

I cant use PXE.

PXE-E79: NBP IS TOO BIG TO FIT IN FREE BASE MEMORY.

I found the tftp logs, client fetch pxeboot, but cant next.

sko · Sep 6, 2021

I once (~2017) built an infrastructure and images for unattended installation of TrueOS clients (when it was based on FreeBSD). I only had to add the MAC address to a whitelist (for DHCP and tftp), and everything was set up automagically via pc-sysintstall. Everything after the basic OS setup was handled via Ansible.

This worked fine on _some_ clients - but PXE is (was?) severely broken in most client/desktop PCs. ACER was especially annoying and broken, but they also often only boot from one specific USB port or have other stupidities like automatically changing the boot order...
By far the most reliable and hassle-free ones were genuine Intel NUCs. But even their variants from e.g. ASRock ('beebox') also had/have annoyingly broken UEFI and didn't work more often than they did. Several variants and generations of ASRock NUCs were just freezing on reboot and could only be resurrected by hard shutdown (pressing powerbutton for >5sec) or unplugging the PSU.

So basically the biggest hurdle is the client itself. Make sure they are actually capable of booting via PXE - just because the vendor puts 'PXE' on the box and as an option in the BIOS/UEFI doesn't tell anything!
I found some versions of the loader to be quite picky via PXE and hang on some systems. Back then IIRC I ended up using an old loader from FreeBSD 11.3 (TrueOS was based on 12-CURRENT back then) as it was the most reliable. Thankfully it (mostly) doesn't really matter what version the loader is and what version of FreeBSD you want to install, as it will just pull the tarballs and splat them on the disk.

This is what I have in the README of that project, regarding the PXE-booting and the spaghetti-staging that TrueOS used back then (thanks to openRC that was heavily unreliable and I replaced a lot of that OpenRC-based staging in the installer with one shellscript and ansible. (All of this shouldn't bother you with a proper FreeBSD install)

Code:

### boot process (includes TrueOS specific quirks):

- BIOS/PXE gets IP, filename, next-server and root-path from DHCP
- BIOS/PXE loads $filename (=pxeboot or loader.efi) via tftp
- pxeboot / loader.efi
        acts as a pre-loader; mounts $root-path from $next-server via NFS and hands over to the standard /boot/loader
- (some basic stuff happens and finally runs rc)
- /etc/rc 
        runs /etc/trueos-rc.sh, then hands over to openrc (runlevels sysinit -> boot -> default)
        - /etc/trueos-rc.sh 
                checks for /dist/TRUEOS.ufs.uzip live-image, fires /etc/SetupLive.sh or /etc/SetupInstall.sh accordingly, then extracts etc/var/root-dist.txz to memfs for the installer environment
                - /etc/SetupInstall.sh
                        checks for /boot/pc-autoinstall.conf and /boot/pc-sysinstall.conf, copies them to /tmp for pc-sysinstall to pick up
        - openrc runlevels (/etc/runlevels/)
                sysinit is empty
                boot: basic system configuration; nothing specific for installation to see here
                default:
                        -> /etc/init.d/local starts everything in /etc/local.d/ (empty)
                        -> /etc/init.d/netmount mounts network shares according to fstab (-> nothing is done here)
- /root/TrueOSStart.sh (extracted from /uzip/root-dist.txz) is fired, starting the actual installation


/root/TrueOSStart.sh has a bug in that it searches for pc-autoinstall.conf and pc-sysinstall.conf in the filesystem root; not in /tmp where /etc/SetupInstall.sh copied it to:
https://github.com/trueos/trueos-core/issues/1497
https://github.com/trueos/trueos-core/pull/1496

UPDATE 2017-11-17: fix has been merged to master

pc-sysinstall makes the hard-coded assumption to find pc-sysinstall.cfg in /tmp, so the renaming from conf to cfg during copy by SetupInstall.sh is essential



#### pc-sysinstall.conf caveats

run setup; right before commiting everything to disk, copy configuration to USB drive and use it as template

packageType=pkg
installMedium=local
localPath=/dist
distFiles=base doc kernel lib32

disk0-part=ZFS 0 [....]   size of 0 causes the zfs slice to get all disk space available (minus efi, swap)


### PXE/NFS setup caveats

script pxe_prep.sh takes a release description and path to local iso file and generates a new pxe bootable dataset with all necessary additions/changes.
All additional files are in git repository @ git.gassner.lan:/var/git/clients/pxe.git releases are managed in seperate branches (ssh keys in repo are for user clients@stor1)

datasets:
/tftpboot/TrueOS-<release> contains the TrueOS-DVD iso (extracted) with the following additions/modifications:
/*      - FreeBSD "pxeboot" or "loader.efi" (trueos version is not mounting NFS?) */  # since TrueOS 17.12 the provided loader by TrueOS correctly boots via PXE
        - /boot/loader.conf (remove vfs.root.mountfrom; NFS mountpoint is received via DHCP)  # mainly cosmetic, after failing to mount a cd-rom loader will proceed with NFS mount; may fail if a cdrom is present!
        - /boot/pc-autoinstall.conf
        - /boot/pc-sysinstall.conf
        - /boot/init_gassner.sh script (called post-installation through pc-sysinstall.conf)
/*      - fixed SetupInstall.sh and TrueOSStart.sh scripts (https://github.com/trueos/trueos-core/pull/1496) in /etc (/uzip/etc-dist.txz) and /root (/uzip/root-dist.txz)
      -> fix has been merged to master on 2017/11/17; should be included in next release */  # included upstream since release 17.12
        - additional packages (ansible) in dist/packages/All + updated package library (see below)

PXE-booting and unattended installation is (was?) a bit quirky, but mainly it is relatively transparent, so you can get it working step by step. But by far the biggest and most common show-stopper is broken PXE implementations on client systems...

msplsh · Sep 7, 2021

You're going to have to create your own ISO to create an "automated" installation of FreeBSD that requires no user intervention. There's no manual for that. I have PXE booted the install discs before, but that just loads the whole thing into RAM over the network. Slow, dumb, but effective.

You can just PXE boot bootonly.iso or mfsBSD (This is a good project) and modify those ISOs to get your automated scripts to work. Figuring out how to generate them is left as an exercise to the reader...

astyle · Sep 7, 2021

PXE and automated installs are a Cute Idea, but over-relying on automation is a dangerous, thorny and slippery path. If there's an error somewhere along the way, the whole house of cards just collapses, and you spend hours trying to figure out where the mistake was. Those same hours might have been better spent setting the stuff up using the inelegant (but reliable) way. If all that is acceptable as part of the territory to you - as msplsh pointed out, it can be a good project.

Oh, and

fr33bsd said:
But reaching sysinstall(8) ended up in a dead end:

Sysinstall has been deprecated for awhile (since FreeBSD 9.0-RELEASE). These days, it's bsdinstall(8).

MG · Apr 27, 2023

Looking for a similar solution. Also, I can't find what's supposed to be the most minimal clear tftpd server configuration for a recent FreeBSD version. I did this before, a long time ago but it got more complicated and depending. Now, it doesn't start but doesn't give an error either... A summary of which files have to be where would be nice.

astyle · Apr 27, 2023

MG said:
Looking for a similar solution. Also, I can't find what's supposed to be the most minimal clear tftpd server configuration for a recent FreeBSD version. I did this before, a long time ago but it got more complicated and depending. Now, it doesn't start but doesn't give an error either... A summary of which files have to be where would be nice.

Well, net/utftpd is in ports, you could give it a try. That port has no maintainer at this time...

tftpd is not in base, AFAIK, but SSHd is... So, in a pinch, you can use SSHd to move files.

MG · Apr 27, 2023

astyle said:
Well, net/utftpd is in ports, you could give it a try. That port has no maintainer at this time...

tftpd is not in base, AFAIK, but SSHd is... So, in a pinch, you can use SSHd to move files.

It's in base (/usr/libexec) and I believe the 13.2 version can't run standalone but needs to be started by inetd. Not sure why, probably security reasons. I see no technical issue that makes in impossible to give the tftpd program the full implementation of whatever is required to make it work. We're root anyway and it's no part of the kernel. Everything should be possible.

blarf · Jun 5, 2023

astyle said:
PXE and automated installs are a Cute Idea, but over-relying on automation is a dangerous, thorny and slippery path. If there's an error somewhere along the way, the whole house of cards just collapses, and you spend hours trying to figure out where the mistake was. Those same hours might have been better spent setting the stuff up using the inelegant (but reliable) way.

What? No.

As it turns out computers are really good at following instructions. Automating things ensures you have a repeatable, verifiable procedure. Doing things by hand ensures you'll eventually fat finger something and spend hours trying to figure out where the mistake was.

astyle said:
Well, net/utftpd is in ports, you could give it a try. That port has no maintainer at this time...

tftpd is not in base, AFAIK, but SSHd is... So, in a pinch, you can use SSHd to move files.

What? No.

Code:

$ whereis tftpd
tftpd: /usr/libexec/tftpd /usr/share/man/man8/tftpd.8.gz /usr/src/libexec/tftpd

By virtue of being part of the base system tftpd is actively maintained. None of the bootloader stuff be it pxeboot, EFI, whatever, supports SSH. If the goal is unattended, automated installations SSH gets you no closer to a solution.

MG said:
It's in base (/usr/libexec) and I believe the 13.2 version can't run standalone but needs to be started by inetd. Not sure why, probably security reasons. I see no technical issue that makes in impossible to give the tftpd program the full implementation of whatever is required to make it work. We're root anyway and it's no part of the kernel. Everything should be possible.

tftpd's architecture hasn't changed substantively since BSD4.2 (so roughly forty years) so if you want to use tftpd, enable it in your inetd config and start inetd.

The only hardcoded path in pxeboot is /boot/loader.rc and that's loaded relative to the configured root path. The pxeboot man page goes into more detail about what your options are for the root path as well as how to configure them. Another officially supported way to load a bsdinstall config file is by building a custom disk image.

For 13.0-RELEASE, bsdinstall and its man page were updated with known limitations and an example of how to build install media. Pretty much everything else (including what you'll find on freebsd.org) is wildly out of date.

MG · Jun 5, 2023

blarf said:
tftpd's architecture hasn't changed substantively since BSD4.2 (so roughly forty years) so if you want to use tftpd, enable it in your inetd config and start inetd.

The only hardcoded path in pxeboot is /boot/loader.rc and that's loaded relative to the configured root path. The pxeboot man page goes into more detail about what your options are for the root path as well as how to configure them. Another officially supported way to load a bsdinstall config file is by building a custom disk image.

For 13.0-RELEASE, bsdinstall and its man page were updated with known limitations and an example of how to build install media. Pretty much everything else (including what you'll find on freebsd.org) is wildly out of date.

I have it now running inside a chrootdir with copies of all binary dependencies. Thing is, I know how to use if officially but this system doesn't use inetd and rc.d at all. Kernel, /etc/rc and login, that's it. I was wondering how a program, regardless of it's age, has to depend on a post-kernel configuration framework, like mandatory. What is it that I can't execute manually but only automated via configuation files. Tftpd returns with non-zero, but without error message if you try that.

astyle · Jun 5, 2023

blarf said:
As it turns out computers are really good at following instructions. Automating things ensures you have a repeatable, verifiable procedure. Doing things by hand ensures you'll eventually fat finger something and spend hours trying to figure out where the mistake was.

Automation only works when procedure components are lined up. One machine may have an AMD GPU, the next one may have an NVidia GPU, the next one has integrated Intel graphics... Trying to account for all kinds of contingencies, and scanning for variants makes automation impractical at best. You have to have a pretty big set of identical hardware, and an ironed-out networking stack before threading the needle with automation. If you have that, then a good template that covers the entire set of hardware copies is next. Software is only half the solution, there's still hardware to consider, y'know.