How to install FreeBSD (/w ZFS) on a remote server from a Linux rescue system

Preface

As many might have already discovered, there are myriads of hosting providers out there that only offer various Linux distributions for (pre-)installation on their servers, but often won't even offer any support for installing any BSD variant.
However, installing FreeBSD from a Linux rescue system - or basically _any_ (rescue) system that allows you to copy a diskimage to a hard drive on your server - really isn't that hard.

There are quite some how-tos and blog postings out there on how to achieve this and even the FreeBSD handbook has a page on this Topic. However, at least all of those I sifted through, were quite a bit outdated, almost none even covered ZFS or only in obscure and complicated ways (been there, done that...) and some were just wrong - at least for more recent FreeBSD versions.
Overall - most of the instructions were overly complicated and it seemed that some authors were trying their best to avoid any shortcut or automatism e.g. by leveraging the FreeBSD installer to do the heavy lifting.

I recently evaluated a working procedure for an upcoming deployment of FreeBSD 11.1-RELEASE on a new root-server without IPMI and remote access only via a Linux rescue system. This how to is directly derived from my notes/documentation of the process.
As every Perl coder knows: TIMTOWTDI - the procedure I will describe was - for me - the most painless and easiest way condensed from various older Instructions, manual pages, personal experiences and quite a lot of 'try & error'.
What I have come up with will very likely not be the most perfect way to do it - so feel free to reply with improvements or (most importantly) corrections on any error I might have made.


Warnings

As stated before, there are many outdated how-tos out in the wild from way back when e.g. 9.0-RELEASE was the latest release. The following procedure has been worked out and tested with 11.1-RELEASE. Don't expect any of the following to work in a few years time with something like a 13.0-RELEASE or even 12.0-RELEASE and even expect minor variations for other dot-Releases!
I'll try to keep these instructions updated for some time, but you have been warned.


Overview

The whole Procedure basically consists of only 3 Steps:
- build a mfsBSD image
- use the Linux Rescue System to write mfsBSD to a disk on the server
- boot mfsBSD and use bsdinstall(8) to do all the dirty work

Especially on the first and last step, however, I discovered some pitfalls that have to be circumvented.


Step 1 - built the mfsBSD image

mfsBSD is basically only a set of scripts to build a stripped-down FreeBSD image that will from memory.
On the project website at http://mfsbsd.vx.sk/ one could download pre-built images, but they are pre-configured for DHCP, which rules them out for most hosting environments.
Additionally we'd really like to use a ssh-key for login instead of exposing an sshd with enabled root-login and a (very weak) password to the whole world.
I've seen boxes being hit with automated bruteforce login attempts to ssh just seconds after they were first deployed - this is a real threat, so don't take this lightly!

To build our own mfsBSD-image we need to clone the mfsBSD github repository:
Code:
$ git clone https://github.com/mmatuska/mfsbsd.git

Now we have a look at the mfsbsd/conf directory. All files with their names ending in .sample contain information on how/what can be configured, but mainly they are just what you would expect on a running FreeBSD system. The .sample-files will be ignored at build time, so copy any file you need and want to edit, stripping the .sample from its name.
At least add your/a working public ssh-key to authorized_keys and add the configuration for one of the available Network cards on your server to either interfaces.conf or rc.conf.
The interfaces.conf file is special, in that it allows you to configure a network adapter without knowing what driver it uses - you only need to know the MAC-address.
interfaces.conf.sample contains all the information you need. A working configuration might look as follows:
Code:
mac_interface="ext1"
ifconfig_ext1_mac="d2:b9:2e:5a:a0:dc"
ifconfig_ext1="inet 10.50.51.169/24"

Make sure to also include rc.conf with sshd_enable="YES" and the defaultrouter set! A working resolv.conf should also be included.

Now you should have the following files in your conf/ directory, with the interfaces.conf being optional:
Code:
$ ls ~/mfsbsd/conf | grep -v sample
authorized_keys
interfaces.conf
loader.conf
rc.conf

To build the image, mfsBSD needs the release-files and sources for the version we want to build our image from. This is most likely the same release we want to install on the target system.
Download the files into a known location (e.g. a separate folder within the mfsbsd directory) and unpack the src.txz file.
Code:
$ mkdir ~/mfsbsd/dist-files
$ cd ~/mfsbsd/dist-files
$ curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/base.txz
$ curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/kernel.txz
$ curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/MANIFEST
$ curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/src.txz
$ tar xzf src

mfsBSD also needs a statically linked version for pkg. This can be placed either into the mfsbsd/tools directory or we can just use the one provided on the system we are building mfsBSD on. I will use the second approach in the example.
Beware of possible ABI incompatibilities when building on a different branch than the resulting mfsBSD image will be! I was running all mfsBSD builds during my tests on a 12.0-CURRENT system and the resulting build for 11.1-RELEASE worked, but you have been warned!

For building mfsBSD we need to elevate to root and run make within the mfsbsd directory, including the paths to BASE, SRC_DIR and PKG_STATIC and increasing the MFSROOT_MAXSIZE.
Code:
# make BASE=/usr/home/user/sko/mfsbsd/dist-files SRC_DIR=/usr/home/user/sko/mfsbsd/dist-files/usr/src PKG_STATIC=/usr/local/sbin/pkg-static MFSROOT_MAXSIZE=100m
Extracting base and kernel ... done
Removing selected files from distribution ... done
Installing configuration scripts and files ... done
Generating SSH host keys ... done
Configuring boot environment ... done
Installing pkgng ... done
Compressing usr ... done
Creating and compressing mfsroot ... done

The mfsBSD image has been created in mfsbsd/mfsbsd-11.1-RELEASE-amd64.img and is ready to be deployed. We can now drop root privileges and move on.


Step 2 - transfer the mfsBSD image to your server and write it to disk

Well, there isn't really much to say about this step - we really only need to touch the Linux system for a minimal amount of time, using only 2 tools everyone is already familiar with: scp and dd.
Just transfer the image over to the Linux rescue system via scp and write it to a bootable (most likely the first) disk.
Usually Linux will name SATA/SAS/SCSI disks "sdN" with N=a being the first disk, N=c the 3rd and N=z the 26nd disk. Virtual disks are named "vdN" in case you are installing on a VPS.

So to dd the image to the first disk:
dd if=mfsbsd-11.1-RELEASE-amd64.img of=/dev/sda bs=1M


Step 3 - boot the mfsBSD image and install FreeBSD

After writing the mfsbsd-image to disk, just reboot the box:
shutdown -r now
Then wait for it to come back online, this time already running FreeBSD. You can now log as root with the ssh-key matching the public key in the authorized_keys file used to build mfsbsd.
Code:
$ ssh root@10.50.51.169
FreeBSD 11.1-RELEASE (GENERIC) #0 r321309: Fri Jul 21 02:08:28 UTC 2017

Welcome to mfsBSD, the memory based FreeBSD distribution.

This is a stripped-down version of FreeBSD without:
- manual pages, info pages, examples
- include files, static library files, development tools
- bind binaries (host, dig, named, etc.)

Feel free to email me with any bug reports or feature suggestions.
Martin Matuska <mm@FreeBSD.org>
http://mfsbsd.vx.sk/
root@mfsbsd:~ #

Before we can actually run bsdinstall(8) we need to create the directory where bsdinstall wants to download additional release files to, as this isn't present on our mfsbsd system.
Somehow bsdinstall on mfsbsd also fails to store the files at this location - so we download all of them before running bsdinstall. To do this we also need to install curl (or wget).
Code:
# mkdir /usr/freebsd-dist
# pkg install curl
# curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/kernel.txz
# curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/base.txz
# curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/MANIFEST
# curl -O https://download.freebsd.org/ftp/releases/amd64/11.1-RELEASE/src.txc
# bsdinstall

Now you can walk through the installation process as usual. As mfsbsd is completely running from memory filesystem (hence the "mfs" in its name), you can use your local disks just as you would on any installation via IPMI or on a local console.

The process of installing FreeBSD is covered in detail by the FreeBSD Handbook. Therefore, and because I strongly believe anyone putting a server out in the wild should be already familiar with installing, setting up and securing a system, I will not cover any of these topics here.

Before rebooting the newly installed system make sure to add at least one user and enable sshd, so you will be able to access the newly installed system. Double-checking rc.conf and resolv.conf might also be a good idea, just to make sure the new system is able to get on the network and resolve names.


Final Words

This is really all that's necessary nowadays to remotely install FreeBSD from any rescue system. No manual setup of ZFS-pools and -datasets, extracting the base system, placing the bootloader on disk or building world in a chroot. These skills are quite handy when things go wrong, but things done manually are prone to human error.
The FreeBSD installer is really up to the task of most if not all scenarios (at least where you won't have proper IPMI...) and makes installation so much more easy, fast and somewhat safer. So why not honor the hard work of the authors of bsdinstall and just use it for what it was meant to do and does well :).


As said in the Preface: This procedure will most likely not be perfect - feel free to reply with any improvements or corrections.

Thanks,
sko
 
Since you downloaded the core FreeBSD release archives to create the mfsBSD image could you not add them to the image at that time instead of re-downloading them from within the running mfsBSD?

It would double the size of the mfsBSD (which might be an issue) but save re-downloading them when you install (which could be a big deal if you need to do this several times).

Do you actually need the src.txz included in the mfsBSD? I thought that contains the source-code to FreeBSD which we don't need if we are doing a binary install. (am I wrong?)

There is probably an easy way to get mfsBSD to start the installer right away. I don't know how, but it would be nice to include in the tutorial.
 
Since you downloaded the core FreeBSD release archives to create the mfsBSD image could you not add them to the image at that time instead of re-downloading them from within the running mfsBSD?

This should be possible and I also thought about it, however, almost always the available local upload bandwidth is only a (tiny) fraction of what is available between a colocation/datacenter and one of the FreeBSD mirrors. So uploading the additional ~180MB (or ~300MB if src.txz should also be included) would take _much_ longer than just downloading them from the nearest/fastest mirror to the server at the datacenter.

Do you actually need the src.txz included in the mfsBSD? I thought that contains the source-code to FreeBSD which we don't need if we are doing a binary install. (am I wrong?)
The Makefile will default to the local /usr/src and 'make' complains if it is empty - I really haven't looked into it any further and just provided 'make' with the sources fitting the release I'm trying to build for to be on the safe side.

There is probably an easy way to get mfsBSD to start the installer right away. I don't know how, but it would be nice to include in the tutorial.
Unfortunately, just creating /usr/freebsd-dist/ isn't sufficient. The installer downloads base.txz but then fails because of the missing MANIFEST file in /usr/freebsd-dist. Only providing the MANIFEST also won't be enough though - after the installer downloaded base.txz and kernel.txz, /usr/freebsd-dist/ still only contains the MANIFEST but none of the downloaded archives. I really haven't checked if this is a bug in bsdinstall, because of some missing permission to the memory file system or just a missing sacrificial chicken.

Actually it should be possible to completely automate the installation, at least after booting into mfsBSD. For one, mfsBSD already provides a 'zfsinstall' script (which I haven't looked into any further :oops:), or one could just use one of the more common/widespread solutions by either using the scripting capabilities of bsdinstall; by including e.g. the puppet or chef package and a configuration in the mfsBSD image or just by using Ansible to connect to mfsBSD and fire bsdinstall with a templated script.
Including additional packages in the image is as easy as copying the according .tbz files into the packages/ directory before building the image - I could have included curl this way (note to self...)

I'm already using Ansible within our infrastructure, but as this was a rather unique task I haven't (yet) tried to use Ansible for the initial deployment. If I find the time I might further look into this and even write some playbooks for this task.


Thanks for your input - I'll surely have a look at some of my workarounds and their causes, and if possible provide some proper solutions.
As for fully/mostly automating the deployment, I think this would better be covered by a seperate how-to or by just releasing the scripts and/or ansible playbooks and roles with some documentation on github.
 
Back
Top