[UEFI/GPT] [Dual-Boot] How to install FreeBSD (with ZFS) alongside another OS (sharing the same disk)

P

patovm04

Important notes:
1) This tutorial assumes you have the OS you want to dual-boot with already installed on your drive, and that you already have freed up some disk space. Essentially, you will be installing FreeBSD with root-on-ZFS on the remaining free space of the disk, instead of using the entire drive as it's done by default by FreeBSD's installer.
2) This method only works for GPT disk layouts using UEFI mode. Legacy BIOS and MBR disks are not supported. As always, make sure to disable Secure Boot and Fast Boot.
3) Here, we use ada0 (disk 1) for everything, and we assume your disk already has an EFI partition, located in partition 1 (ada0p1). Change it according to your setup, if needed. Run gpart show to be sure.
4) We will create our ZFS Pool with "Stripe" as the Virtual Device type (vdev).
5) After completing installation, to be able to choose what OS to boot you'll need to either install rEFInd boot manager, or use GRUB (assuming the other OS is a Linux distro). I personally recommend rEFInd, but I won't detail how to install it here. Neither will I explain how to alter the GRUB config. Just gonna show you how the respective entries should look like in each case.
6) Finally, I'm not responsible for any problem or data loss you may experience. This tutorial is not an unsafe procedure if you understand what you're doing (especially in regards to selecting the correct disk where you want to install). Anyway, do it at your own risk!


Well, let's start:
First of all, boot your FreeBSD install USB stick. Proceed installing as usual, until you reach the "Partitioning" stage. Here, choose the "Shell" option, so you can create your swap and ZFS partitions by hand:

# Load the ZFS kernel module:
kldload zfs

# Force 4K sectors:
sysctl vfs.zfs.min_auto_ashift=12

# Create a swap partition (in this case, of 4 GB size):
gpart add -a 4k -l swap0 -s 4G -t freebsd-swap ada0

# Create a ZFS partition to fill the remaining free space:
gpart add -a 4k -l zfs0 -t freebsd-zfs ada0

# Mount tmpfs in /mnt, where the installer will install the operating system to:
mount -t tmpfs tmpfs /mnt

# Create your zroot pool:
zpool create -f -o altroot=/mnt -O compress=lz4 -O atime=off -m none zroot /dev/gpt/zfs0

# Create ZFS file system hierarchy:
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default

# Define the default ZFS datasets for root zpool:
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp # We'll be using tmpfs for /tmp.
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create zroot/usr/home
zfs create -o setuid=off zroot/usr/ports
zfs create zroot/usr/src
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs create -o setuid=off zroot/var/tmp

# Create a symlink to /usr/home:
ln -s /usr/home /mnt/home

# Change /tmp and /var/tmp permissions:
chmod 1777 /mnt/tmp # Not needed now.
chmod 1777 /mnt/var/tmp

# Configure the Boot Environment:
zpool set bootfs=zroot/ROOT/default zroot

# Instruct FreeBSD to mount ZFS pools during system initialization:
printf 'zfs_enable="YES"' >> /tmp/bsdinstall_etc/rc.conf

# Add an fstab entry to use a GELI-encrypted swap partition:
printf "/dev/gpt/swap0.eli\tnone\tswap\tsw\t0\t0\n" >> /tmp/bsdinstall_etc/fstab

# Add an fstab entry to use tmpfs for /tmp:
printf "tmpfs\t/tmp\ttmpfs\trw,mode=1777\t0\t0\n" >> /tmp/bsdinstall_etc/fstab

# Mount the existing EFI partition of your disk:
mount -t msdosfs /dev/ada0p1 /media

# Create FreeBSD boot directory in the EFI partition:
mkdir -p /media/efi/freebsd

# Copy FreeBSD loader to the previously created directory:
cp /boot/loader.efi /media/efi/freebsd/loader.efi

# Now add a UEFI boot entry:
efibootmgr --create --activate --label "FreeBSD" --loader "/media/efi/freebsd/loader.efi"

# Unmount EFI partition:
umount /media

# Exit Shell Partitioning mode, so bsdinstall can continue and complete the installation:
exit


After finishing the installation, don't forget to add a boot entry for FreeBSD in GRUB or rEFInd.
Here I share some sample entries to guide you a bit. Remember to modify them to reflect your own reality:

# For GRUB:
Code: 
menuentry "FreeBSD" {
    insmod part_gpt
    insmod fat
    set root=(hd0,gpt1)
    chainloader /efi/freebsd/loader.efi
}

# For rEFInd, add something like this at the end of refind.conf:
Code: 
# FreeBSD
menuentry "FreeBSD" {
    loader /efi/freebsd/loader.efi
    icon /efi/refind/icons/os_freebsd.png
}

That's all. Hope you find it useful!

Sources:
https://www.freebsd.org/doc/handbook/bsdinstall-partitioning.html
https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
http://kev009.com/wp/2016/07/freebsd-uefi-root-on-zfs-and-windows-dual-boot/
https://github.com/freebsd/freebsd-src/blob/releng/13.0/usr.sbin/bsdinstall/scripts/zfsboot
https://github.com/freebsd/freebsd-src/blob/releng/13.0/usr.sbin/bsdinstall/scripts/bootconfig


EDIT 1:
-Added some sources.

EDIT 2:
-Reworded the title for better clarity.
-Added a note about FreeBSD's efi loader name used in this tutorial.
-Removed /tmp from ZFS datasets, to use a tmpfs /tmp instead. This way it provides better performance and doesn't cause troubles with some programs, like Firefox on Wayland, for instance.

UPDATE (2021/02/12):
-Fixed some little issues.
-Added useful efibootmgr command.
-Updated FreeBSD efi boot loader name and location to match new FreeBSD 13.0 defaults.
-Added more sources.
 
Last edited:
A

amnixed

patovm04 said:
Hello patovm04,
Click to expand...
I ran into a couple of problems following your guide:
patovm04 said:
# Instruct FreeBSD to mount ZFS pools during system initialization:
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
Click to expand...
# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory

mkdir /mnt/etc or create zfs filesystem for /etc?

patovm04 said:
# Mount the existing EFI partition of your disk:
mount -t msdosfs /dev/ada0p1 /media
Click to expand...
gpart(8) says ada0p1 is the type ms-recovery (NTFS?) and mount(8) fails. Does the Windows 10 installer even have the option for FAT (=msdosfs?)?
 
OP
P

patovm04

twllnbrck said:
Which advantage do you have to install FreeBSD with root on zfs when you dual boot from the same disk? Im just wondering ..
Click to expand...
To be able to take advantage of ZFS most useful features such as boot environments, as far as I understand.
 
OP
P

patovm04

amnixed said:
I ran into a couple of problems following your guide:

# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory

mkdir /mnt/etc or create zfs filesystem for /etc?


gpart(8) says ada0p1 is the type ms-recovery (NTFS?) and mount(8) fails. Does the Windows 10 installer even have the option for FAT (=msdosfs?)?
Click to expand...
Hmm it's difficult to help you if you're not following the steps to the letter (for instance, you are running bash which isn't even part of FreeBSD's live install media).
Also it's uncommon EFI not to be your first partition, but given that's your case, it's just a matter of figuring out the correct partition number (/dev/ada0p3 for example).
 
A

amnixed

patovm04 said:
Hmm it's difficult to help you if you're not following the steps to the letter (for instance, you are running bash which isn't even part of FreeBSD's live install media).
Also it's uncommon EFI not to be your first partition, but given that's your case, it's just a matter of figuring out the correct partition number (/dev/ada0p3 for example).
Click to expand...
You are right, I was probably inattentive following your guide (which is actually excellent), gave up too soon and proceeded with an MBR/UFS installation. Will try it again!
 
3

313

Thanks to the OP for the excellent tutorial. Following the instructions I was able to get dual boot going without any problems. I did however notice a couple of things that need correction, should one follow the steps by the letter.

1) The following is conflicting:

# Mount a tmpfs to /mnt, where the installer will install the operating system into:
mount -t tmpfs tmpfs /mnt
...
# Mount your pool:
mount -t zfs zroot/ROOT/default /mnt
Click to expand...

If tmpfs is mounted on /mnt, the latter mount command causes the 'device busy' error that some users noticed. The tmpfs mount command should be skipped.

2) This does not work as it is, because /mnt/etc does not exist:

# Instruct FreeBSD to mount ZFS pools during system initialization:
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
Click to expand...

Instead, I did
echo 'zfs_enable="YES"' >> /tmp/bsdinstall_etc/rc.conf
..and the results were as expected.
 
OP
P

patovm04

313 said:
Thanks to the OP for the excellent tutorial. Following the instructions I was able to get dual boot going without any problems. I did however notice a couple of things that need correction, should one follow the steps by the letter.

1) The following is conflicting:



If tmpfs is mounted on /mnt, the latter mount command causes the 'device busy' error that some users noticed. The tmpfs mount command should be skipped.

2) This does not work as it is, because /mnt/etc does not exist:



Instead, I did
echo 'zfs_enable="YES"' >> /tmp/bsdinstall_etc/rc.conf
..and the results were as expected.
Click to expand...
Hmm interesting 🤔, as I did follow it to the letter before posting it here and didn't have those issues. Maybe something's changed since then... If I recall correctly I used FreeBSD 12.1-RELEASE at the time.
Well, I guess the how-to needs some updating now. Gonna do it some time in the future after checking it thoroughly. Thank you
 
Zamana

Zamana

Hi!

Can you tell me if your guide would work in the case of 2 separated HDDs?

On my laptop, I already have Windows 10 installed in the first HDD (for work), and I would like to install FreeBSD in the second HDD (for personal use).

Thanks.
 
drewlander

drewlander

Thank you for the writeup! Well done!

Just for anyone else who runs into this:
I also ran into:
Bash: 
# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory
making the directory then re-running worked for me

After copying boot1.efi, I made a directory in EFI specifically for FreeBSD then ran:
Bash: 
 efibootmgr -c -l /mnt/EFI/FREEBSD/BOOTX64.EFI -L FREEBSD

working like a charm
 
OP
P

patovm04

drewlander said:
Thank you for the writeup! Well done!

Just for anyone else who runs into this:
I also ran into:
Bash: 
# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory
making the directory then re-running worked for me

After copying boot1.efi, I made a directory in EFI specifically for FreeBSD then ran:
Bash: 
 efibootmgr -c -l /mnt/EFI/FREEBSD/BOOTX64.EFI -L FREEBSD

working like a charm
Click to expand...
Thanks, I just updated the guide. Those issues should be solved now :)
 
OP
P

patovm04

Zamana said:
Hi!

Can you tell me if your guide would work in the case of 2 separated HDDs?

On my laptop, I already have Windows 10 installed in the first HDD (for work), and I would like to install FreeBSD in the second HDD (for personal use).

Thanks.
Click to expand...
Not sure, I've never done it to be honest...
 
S

scottro

I just did this on FreeBSD-13.0-BETA2. I use CentOS-8-stream grub to boot it. (Though I had to lock grub at an earlier version as later versions weren't booting FreeBSD, though I had a different custom.cfg, as that was on a UFS partition).
Anyway, this time, got an error that it couldn't find the file, hit any key to continue, which I did, and it booted FreeBSD properly. So, anyway, thanks again, I think this is kind of neat to have a ZFS partition dual booting with Linux. It does, aside from giving people the chance to use bectl or beadm, also give them a chance to experience ZFS stability and such, though in laptop use, I don't know if it will be noticeable.
 
S

scottro

For the sake of accuracy, I eventually figured this out. I'd put insmod part gpt (a space, rather than an underline) so the error was that it couldn't find part.mod. As I mentioned if I hit any key to continue, it still worked and if I eliminated the insmod line for part_gpt it worked without giving an error. However, once I figured out what I'd done wrong, I corrected the part<space>gpt to read part_gpt and then all was good. Thanks again for this tutorial.
 
S

scottro

Our of curiosity, I tried this with 14-CURRENT. (One nice thing about CURRENT on my hi res yoga2, is that it doesn't seem to need the drm-kmod--even during installation the fonts are nice and readable.

However, with CURRENT the method doesn't work. Instead, when I choose shell at the partitioning section, I get an error that it's unable to create, The exact error is
Code: 
Could not create directory /mnt/user/freebsd-dist
Then goes on to say an installation step has failed.
This is a CURRENT snapshot from 2021-03-14. I realize that mentioning CURRENT is more or less taboo, but as I'd be installing it in lieu of Linux, it is also an I like FreeBSD instead of Linux thread, and they can go on for pages.

I'm wondering if it can be worked around by dropping to shell right at installation, but my knowledge of ZFS is far less than yours, and I'm not confident of what commands I'd run. And of course, as it's CURRENT, the next snapshot might fix it. One other nice thing about having it on a zfs partition, I realized, is that if something freezes, which sometimes happens on this laptop, and I have to manually power it off, ZFS would be, I think, less likely to have data corrupted. (Though there's nothing important on this particular laptop). \
 
Phishfry

Phishfry

scottro said:
I'm wondering if it can be worked around by dropping to shell right at installation
Click to expand...
There is little that can't be done at the post install shell.
You can even download packages into the new install.
But the pre-install shell has drawbacks. No disk is yet selected and no internet available.
It is good for starting modules like Chelsio cards that need a loader.conf setting for starting up.
The shell during 'disk partition' phase is good for concocting a graid1 array. I could see the same for custom ZFS.
 
S

scottro

Thanks. I also tried setting up a ZFS partition pre-install, but didn't know what I was doing well enough to get it working that way. As it's a CURRENT snapshot, hopefully, I"ll have better luck with the next one.
 
Phishfry

Phishfry

One thing to mention with the bsdinstall 'partition' phase shell.
Notably I had to make some command line settings by hand for my gmirror arrangement.


Code: 
echo "/dev/mirror/gm0p2 / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab
echo 'geom_mirror_load="YES"' >> /tmp/bsdinstall_boot/loader.conf
 
S

scottro

To update, trying again on the latest CURRENT, using a snapshot from 2021-03-18 (yesterday), your method once again works perfectly for me.
I'll also make a comment, that some did ask the purpose of doing this, and your answer is certainly good, that one can take advantage of beadm or bectl. I have also found, on this particular laptop, sometimes the system has frozen, if say, I made a typo in .xinit and sometimes, I have had to shut it by holding down the power switch. While nothing has ever been hurt by that, having something in ZFS makes me less nervous about shutting the laptop down that way, though even if the whole drive was zfs, I wouldn't recommend the method. :)
 
You must log in or register to reply here.
Top