HOWTO: Multi-boot FreeBSD + NetBSD + OpenBSD

[rbranco]

I will have the latest version here:

howtos/BSD/Multiboot.md at master · ricardobranco777/howtos

tips and howtos. Contribute to ricardobranco777/howtos development by creating an account on GitHub.

github.com

Installing NetBSD 10.x & FreeBSD 14.x in multiboot.

## Install NetBSD

Adapted from [NetBSD wiki](https://wiki.netbsd.org/Installation_on_UEFI_systems/):

# Use Korn shell
ksh


dkctl wd0 listwedges
gpt destroy wd0
gpt create wd0
# Use this size to avoid the formatting done as FAT-16
gpt add -l gptboot -t efi -s 512m wd0
gpt add -l NetBSD -t ffs -s 300g wd0
gpt add -l swap -t swap wd0


dkctl wd0 listwedges


newfs_msdos /dev/rdk5
mount -t msdos /dev/dk5 /mnt
mkdir -p /mnt/EFI/boot
cp /usr/mdec/*.efi /mnt/EFI/boot
umount /mnt


newfs -O 2 dk6

## Install FreeBSD

Adapted from [FreeBSD wiki](https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot):

gpart add -a 1m -t freebsd-zfs -l FreeBSD ada0


# Optional: Enable encryption
geli init -g -s 4k gpt/FreeBSD
geli attach gpt/FreeBSD


mount -t tmpfs tmpfs /mnt


# Drop .eli extension if you don't want encrypt
zpool create -o altroot=/mnt zroot gpt/FreeBSD.eli


# Create ZFS pool with transparent compression enabled
zfs set compress=on zroot


zfs create -o mountpoint=none                                  zroot/ROOT
zfs create -o mountpoint=none                                  zroot/ROOT/default
mount -t zfs zroot/ROOT/default /mnt


zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
zfs create -o canmount=off -o mountpoint=/usr zroot/usr
zfs create zroot/home
zfs create -o setuid=off zroot/usr/src
zfs create zroot/usr/obj
zfs create -o mountpoint=/usr/ports -o setuid=off zroot/usr/ports
zfs create -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o canmount=off -o mountpoint=/var zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=on -o setuid=off zroot/var/tmp


chmod 1777 /mnt/var/tmp /mnt/tmp


zpool set bootfs=zroot/ROOT/default zroot


echo /dev/gpt/swap0 none swap sw 0 0 >> /tmp/bsdinstall_etc/fstab


# Backup GELI header
mkdir -m 0750 /mnt/var/backups/
cp -p /var/backups/* /mnt/var/backups/

After installation before reboot:

sysrc zfs_enable="YES"
cat >> /boot/loader.conf << EOF
aesni_load="YES"
geom_eli_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
EOF

Configure Grub on any bootable system. Make sure to populate this partition with the relevant .efi files.

cat > /etc/grub.d/40_custom <<EOF
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry "NetBSD" {
        insmod part_gpt
        insmod chain
        chainloader (hd0,gpt1)/EFI/boot/bootx64.efi
}

menuentry "FreeBSD" {
        insmod part_gpt
        insmod chain
        chainloader (hd0,gpt1)/EFI/freebsd/loader.efi
}
EOF


grub2-mkconfig -o /boot/grub2/grub.cfg

 

[rbranco]

I decided to install both on a Dell Precision 5520. I also planned to install OpenBSD but the fact that it doesn't support TRIM on SSD was a deal-breaker to me. Also OpenIndiana but the stupid ISO doesn't seem to work on this laptop.

 

[T-Daemon]

I like to comment some points of your howto:

geli init -g -s 4k gpt/FreeBSD

If data key length ( -l ) is not specified, it defaults to 128 ( geli(8) ). Unless 128 is set intentional 256 would be a better choice.

256 is also set by the zfsboot installation script.

/var/log/bsdinstall_log

geli init -bg -e AES-XTS -J - -l 256 -s 4096 "nvd0p3"

(side note, setting -e AES-XTS explicitly is unnecessary, the default is AES-XTS anyway)

zpool create -o altroot=/mnt zroot gpt/FreeBSD.eli
...
zfs create -o atime=on -o exec=off -o setuid=off zroot/var/mail

In your pool creation "atime" is on by default, no need to enable explicitly on child dataset, it would be inherited from parent dataset.

The menu guided installation script disables "atime" in the pool creation.

/var/log/bsdinstall_log

zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f "zroot"   nvd0p3.eli

cat >> /boot/loader.conf << EOF
aesni_load="YES"

aesni(4) is built in the GENERIC kernel, no need to load explicitly. What can be loaded is cryptodev(4), that's what the installer script sets.



By the way, you could skip the manual FreeBSD installation and perform a guided installation instead, which is much faster and has the advantage all system configuration is done by utilities (hostname, root password network interface configuration, adding users, etc).

Install first encrypted FreeBSD with a large swap size (set in the "ZFS Configuration" menu), then delete swap later to make space for other OS installations.

Choose the swap size as NetBSD installation size (eventually including other OS installations) + FreeBSD swap.

After FreeBSD installation, before reboot, edit FreeBSD /etc/fstab, change swap device name to GPT label. That should be gpt/swap0 (swap0 label set automatic by the installer script). If swap is encrypted gpt/swap0.eli.

Delete swap before leaving the FreeBSD installer (gpart(8) delete). You need swapoff(8) before delete.

Install NetBSD with space left for FreeBSD swap, copy NetBSD boot code into (FreeBSD created) ESP, create FreeBSD swap, encrypted or without encryption.

In the above method, the FreeBSD partition would be placed at the end of the disk. If you want, it can be moved to any position on the disc, provided the position is not to close to the original partition (read the next paragraph why).

Create a partition with the exact size in blocks of the original (not smaller, not larger), use gpart(8) -b option to specify where the partition should start, dd(1) original partition to new partition, delete original partition.

 

[rbranco]

By the way, you could skip the manual FreeBSD installation and perform a guided installation instead, which is much faster and has the advantage all system configuration is done by utilities (hostname, network interface configuration, adding users, etc).

Install first encrypted FreeBSD with a large swap size (set in the "ZFS Configuration" menu), then delete swap later to make space for other OS installations.

Choose the swap size as NetBSD installation size (eventually including other OS installations) + FreeBSD swap.

After FreeBSD installation, before reboot, edit FreeBSD /etc/fstab, change swap device name to GPT label. That should be gpt/swap0 (swap0 label set automatic by the installer script). If swap is encrypted gpt/swap0.eli.

Delete swap before leaving the FreeBSD installer (gpart(8) delete). You need swapoff(8) before delete.

Install NetBSD with space left for FreeBSD swap, copy NetBSD boot code into (FreeBSD created) ESP, create FreeBSD swap, encrypted or without encryption.

In the above method, the FreeBSD partition would be placed at the end of the disk. If you want, it can be moved to any position on the disc, provided the position is not to close to the original partition (read the next paragraph why).

Create a partition with the exact size in blocks of the original (not smaller, not larger), use gpart(8) -b option to specify where the partition should start, dd(1) original partition to new partition, delete original partition.

This is wickedly clever! You already gave me the perfect excuse to install again because of the AES-256 setting but this is a new level. Thanks a lot!

 

[T-Daemon]

This is wickedly clever! You already gave me the perfect excuse to install again because of the AES-256 setting but this is a new level. Thanks a lot!

Your are welcome.

I had the swap size idea borrowed from review "bsdinstall: Allow to set ZROOT partition size" to dual boot install FreeBSD and a Linux distribution.

I wanted the zroot partition aft the ESP and FreeBSD swap (positioned at the beginning of the disk), and install a Linux distribution at the end of disk.

I'm sure this can be done with a interactive script, but I'm not aware of such a script publicly available, and to write a script, I have to familiarize myself thoroughly with shell scripting, which I'm not inclined to do that at the moment.

Install NetBSD with space left for FreeBSD swap, copy NetBSD boot code into (FreeBSD created) ESP, create FreeBSD swap, encrypted or without encryption.

On second thought, the swap creation step can be done at the end of the initial FreeBSD installation.

 

[rbranco]

Your are welcome.

I had the swap size idea borrowed from review "bsdinstall: Allow to set ZROOT partition size" to dual boot install FreeBSD and a Linux distribution.

I wanted the zroot partition aft the ESP and FreeBSD swap (positioned at the beginning of the disk), and install a Linux distribution at the end of disk.

I'm sure this can be done with a interactive script, but I'm not aware of such a script publicly available, and to write a script, I have to familiarize myself thoroughly with shell scripting, which I'm not inclined to do that at the moment.


On second thought, the swap creation step can be done at the end of the initial FreeBSD installation.

I just finished doing it and it worked wonderfully. I just skipped the part of moving the partition since it was taking some time and it's not really necessary.

I think there's a bug in the installer that makes /zroot a mountpoint for zroot. I had to manually fix this later. When I have time I'll open a bug though I didn't see this in -CURRENT.

For OpenIndiana I'm thinking of doing zfs send/receive from/to a VM.

Maybe also OpenBSD and do a weekly TRIM after mounting its partition from another BSD, WDYT?

 

[rbranco]

Multi-booting FreeBSD, NetBSD & OpenBSD, all encrypted except for NetBSD, is incredibly easy. I updated the guide in Github.

 

[freezmi]

I will have the latest version here:

howtos/BSD/Multiboot.md at master · ricardobranco777/howtos

tips and howtos. Contribute to ricardobranco777/howtos development by creating an account on GitHub.

github.com

first of, thanks for sharing.

I had an old Thinkpad X1 gathering dust so I decided to install all 3 BSDs on one disk.
I tried my best to follow your guide, but NetBSD was having none of UEFI on my hardware, the kernel kept freezing during boot. Once going the BIOS route it's rock solid.

here are my gotchas for a bios with ms-dos partition table multiboot setup:
- there is no need whatsoever to wrangle with large swap partitions and re-use them to install the next OS. I found that all OSes can be told their size and location on the disk (with both msdos and gpt tables). preemptive creation of the OpenBSD partition eases the install a lot, as you noted in your howto. this can be done in the OBSD's installer shell IIRC.
- all BSDs can have the swap as a slice inside their partition
- install one of the BSDs that have a grub2 package (NetBSD or FreeBSD) first. installing grub into the MBR will allow booting all BSDs and the second stage (and grub.cfg) will be available on the BSD partition, thus not wasting an extra primary partition for the boot manager.
if the partition start offset is too big then grub will not be able to boot, so make sure the OS with the grub package is installed first.

# fdisk ada0

******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=969021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=969021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 169 (0xa9),(NetBSD)
start 9207, size 268427016 (131067 Meg), flag 0
beg: cyl 0/ head 146/ sector 10;
end: cyl 1023/ head 97/ sector 27
The data for partition 2 is:
sysid 166 (0xa6),(OpenBSD)
start 268437504, size 268435456 (131072 Meg), flag 0
beg: cyl 1023/ head 254/ sector 63;
end: cyl 1023/ head 254/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 536872960, size 268435456 (131072 Meg), flag 80 (active)
beg: cyl 1023/ head 255/ sector 63;
end: cyl 1023/ head 255/ sector 63
The data for partition 4 is:
<UNUSED>

grub.cfg excerpt, partition/slice UUIDs are specific to my setup:

menuentry 'NetBSD' {
insmod part_bsd
insmod ufs2
search --no-floppy --fs-uuid --set=root 69bcd36c77f5bb57
knetbsd /netbsd -r wd0a -z
knetbsd_module_elf /stand/amd64/10.1/modules/ffs/ffs.kmod
}
menuentry 'OpenBSD' {
insmod chain
#set root=(hd0,msdos2)
search --no-floppy --fs-uuid --set=root 69bce21a5dcc6cdc
chainloader +1
}
menuentry 'FreeBSD' {
insmod ufs2
#set root=(hd0,msdos3,bsd1)
search --no-floppy --fs-uuid --set=root 69bcf416b7938c0c
kfreebsd /boot/loader
}

later edit:
and oh yeah, remove any 'set -e' from the grub configuration templates if grub-mkconfig is not run under bash