How to manually setup FreeBSD on zfs root?

F

Ferocious1881

I have been following this guide https://www.c0ffee.net/blog/freebsd-full-disk-encryption-uefi to setup full disk encryption in my FreeBSD machine, until the part of the root, as I want it to be a zfs filesystem. Thus, in resume, I have wrote the following commands, in the shell partitioning part:
Code: 
gpart show
gpart destroy -F ada0
gpart create -s gpt ada0
gpart add -t efi -l freebsd-efi -a 4k -s 800k ada0
newfs_msdos /dev/gpt/freebsd-efi
mount -t msdosfs /dev/gpt/freebsd-efi /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/boot1.efi /mnt/EFI/BOOT/BOOTX64.efi
echo BOOTx64.efi > /mnt/EFI/BOOT/STARTUP.NSH
umount /mnt
gpart add -t freebsd-ufs -l freebsd-boot -a 4k -s 1g ada0
newfs -U -L bootfs /dev/gpt/freebsd-boot
gpart add -t freebsd-ufs -l freebsd-root -a 4k ada0
geli init -b -e AES-XTS -l 256 -s 4096 /dev/gpt/freebsd-root
geli attach /dev/gpt/freebsd-root
This being the moment where I deattach form the guide and start to setup a zfs root like
Code: 
zpool create rpool /dev/gpt/freebsd-root.eli
zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/mnt -o compression=lz4 rpool/ROOT/default
then I go back to the guide and run
Code: 
mkdir /mnt/bootfs
mount /dev/gpt/freebsd-boot /mnt/bootfs
cd /mnt
mkdir bootfs/boot
ln -s bootfs/boot
add /dev/gpt/freebsd-boot /bootfs ufs rw 0 0 to the fstab file
but have not added anything to the /boot/loader.conf, to configure it latter.
When installation ended I selected the option to open shell to make some final adjusments and /mnt was empty.

Thus have no idea of what went wrong.
 
That article is literally seven years old, and is written with a completely different outcome in mind. Read the ZFS chapter in the handbook.
 
Beastie7 said:
the ZFS chapter in the handbook.
Click to expand...

I sped through that chapter (currently 22), couldn't see what's required. Which section (or subsection)?

I also found a stale wiki page (RootOnZFS) directing readers to the Installing FreeBSD chapter … similarly, I couldn't see what's required.
 
Look at what OP did with their partitioning. Their entire partitioning scheme is jacked. They'll have to re-do it from scratch. You can't create a separate ZFS pool ontop a UFS/GELI formatted scheme, on the same drive. There is no ZFS partition, filesystems for their entire hierarchy weren't created and mounted for zroot, ZFS wasn't enabled in rc.conf, etc.

OP. Read chapters 20/22, and try again.
 
Ferocious1881 said:
I have been following this guide https://www.c0ffee.net/blog/freebsd-full-disk-encryption-uefi to setup full disk encryption in my FreeBSD machine,
Click to expand...
That linked guide is outdated. A separate unencrypted UFS boot partition containing the kernel is not necessary since a long time.

/boot/boot1.efi has been deprecated, now /boot/loader.efi is used, no /EFI/BOOT/STARTUP.NSH is necessary, etc.


I assume, if you followed the guide by the letter, you want the whole disk (besides the EFI System Partition) encrypted root-on-ZFS, (that's how the author partitioned the disk).

If that's the case, don't bother with manual configuration.

The FreeBSD installer can install on whole disks encrypted root-on-ZFS automatically, see handbook, chapter 2.6.4. Guided Partitioning Using Root-on-ZFS (note the "E Encrypt disks?" configuration option, eventually you want to enable "W Encrypt Swap?" as well).


If you want a partial disk FreeBSD installation, so the system can share the disk with another OS, please let us know, so we can suggest a appropriate method or point out a howto.
 
T-Daemon said:
That linked guide is outdated. A separate unencrypted UFS boot partition containing the kernel is not necessary since a long time.

/boot/boot1.efi has been deprecated, now /boot/loader.efi is used, no /EFI/BOOT/STARTUP.NSH is necessary, etc.


I assume, if you followed the guide by the letter, you want the whole disk (besides the EFI System Partition) encrypted root-on-ZFS, (that's how the author partitioned the disk).

If that's the case, don't bother with manual configuration.

The FreeBSD installer can install on whole disks encrypted root-on-ZFS automatically, see handbook, chapter 2.6.4. Guided Partitioning Using Root-on-ZFS (note the "E Encrypt disks?" configuration option, eventually you want to enable "W Encrypt Swap?" as well).


If you want a partial disk FreeBSD installation, so the system can share the disk with another OS, please let us know, so we can suggest a appropriate method or point out a howto.
Click to expand...
Yes, but also want to add some extra ZFS options at root, mainly I want utf8only=off, which cannot be configured with the installer.
 
Ferocious1881 said:
Yes, but also want to add some extra ZFS options at root, mainly I want utf8only=off, which cannot be configured with the installer.
Click to expand...
The default value for "utf8only" is "off", no need to set explicitly.

zfsprops(7)
Rich (BB code): 
     utf8only=on|off
       Indicates whether the file system should reject file names that include
       characters that are not present in the UTF-8 character code set.  If
       this property is explicitly set to off, the normalization property must
       either not be explicitly set or be set to none.  The default value for
       the utf8only property is off.  This property cannot be changed after
       the file system is created.

What other extra options have you in mind to add at the pool or dataset creation? Perhaps those are not necessary to set as well.

Ferocious1881 said:
Trying to follow step one gpart destroy ada0 from a fresh install usb stick, I got Device Busy.
Click to expand...
Try with the -F option.
 
T-Daemon said:
What other extra options have you in mind to add at the pool or dataset creation? Perhaps those are not necessary to set as well.
Click to expand...
I want casesensitivity=casesensitive, xattr=on, which probably are already the default, maybe not the first one, I am not sure If I want native compression, nor if some deduplication, but if so it would be dedup=verify to get a byte per byte check of the same data if deduplication.
atime=off also looks interesting
 
Ferocious1881 said:
I want casesensitivity=casesensitive, xattr=on, which probably are already the default, maybe not the first one, I am not sure If I want native compression, nor if some deduplication, but if so it would be dedup=verify to get a byte per byte check of the same data if deduplication.
atime=off also looks interesting
Click to expand...
As well copies=2, with deduplication enabled.
and rstchown=on, in the /home/ filesystem.
and setuid=off, because that was a very bad idea.
And looks like is all of it, after reading https://docs.oracle.com/cd/E36784_01/html/E36835/gazsd.html#scrolltoc
 
Ferocious1881 said:
I want casesensitivity=casesensitive, xattr=on, which probably are already the default,...
atime=off also looks interesting
Click to expand...
These are default settings on the installer.

For all other non-default installer properties, a scripted installation is an option. See /etc/installerconfig bsdinstall(8)), ZFSBOOT_BOOT_POOL_CREATE_OPTIONS, ZFSBOOT_DATASETS [1].


Or for the lazy, transform the default installer into a custom installer, edit /usr/libexec/bsdinstall/zfsboot on the installation media [1] to your liking:
Bash: 
54 : ${ZFSBOOT_POOL_CREATE_OPTIONS:=-O compress=lz4 -O atime=off}


145 f_isset ZFSBOOT_DATASETS || ZFSBOOT_DATASETS="
146         # DATASET       OPTIONS (comma or space separated; or both)
147
148         # Boot Environment [BE] root and default boot dataset
149         /$ZFSBOOT_BEROOT_NAME                           mountpoint=none
150         /$ZFSBOOT_BEROOT_NAME/$ZFSBOOT_BOOTFS_NAME      mountpoint=/
151
152         # Home directories separated so they are common to all BEs
153         /home           mountpoint=/home
154
155         # Compress /tmp, allow exec but not setuid
156         /tmp            mountpoint=/tmp,exec=on,setuid=off
157
158         # Don't mount /usr so that 'base' files go to the BEROOT
159         /usr            mountpoint=/usr,canmount=off
160
161         # Ports tree
162         /usr/ports      setuid=off
163
164         # Source tree (compressed)
165         /usr/src
166
167         # Create /var and friends
168         /var            mountpoint=/var,canmount=off
169         /var/audit      exec=off,setuid=off
170         /var/crash      exec=off,setuid=off
171         /var/log        exec=off,setuid=off
172         /var/mail       atime=on
173         /var/tmp        setuid=off


[1]

When .img, option one: dd(1) on USB, boot into single-user mode, mount file system rw, edit zfsboot, exit single-user mode. Or mount the USB on another system, edit the file from there.

Option two: attach .img as a memory disk ( mdconfig(8) ), mount file system, edit file, dd(1) on USB stick.

When .iso, extract .iso, edit zfsboot, make iso image. How to, see bsdinstall(8) BUILDING AUTOMATIC INSTALL MEDIA at the end of the manual.

When using /etc/installerconfig, simply add it on the installer media.

Modified .img and .iso files can be tested in a VM before using for real production installation.
 
Cath O'Deray said:
I'd like to know how the device became busy.
Click to expand...

gpart(8)
Rich (BB code): 
   Deleting Partitions and Destroying the Partitioning Scheme
     If a Device busy error is shown when trying to destroy a partition table,
     remember that all of the partitions must be deleted first with the delete
     action.  In this example, da0 has three partitions:

           /sbin/gpart delete -i 3 da0
           /sbin/gpart delete -i 2 da0
           /sbin/gpart delete -i 1 da0
           /sbin/gpart destroy da0

     Rather than deleting each partition and then destroying the partitioning
     scheme, the -F option can be given with destroy to delete all of the
     partitions before destroying the partitioning scheme.  This is equivalent
     to the previous example:

           /sbin/gpart destroy -F da0
 
Beastie7 said:
That article is literally seven years old, and is written with a completely different outcome in mind. Read the ZFS chapter in the handbook.
Click to expand...
Hi,

I've been following the guide for the most part (I want to run UFS), and every time I get stuck at the part where I'm told to write to /etc/fstab. I keep getting the error that this file is on a read-only filesystem (which is true because it's in the ISO).

Can we talk about this? I think I need more help in understanding where I'm going wrong.
 
In general, while you are doing an installation, the target (in other words, where you are installing stuff) is (temporarily) mounted to a different place, for example /mnt. So instead of /etc/fstab, you might be looking for /mnt/etc/fstab. Or wherever target is mounted.
 
Ferocious1881 said:
As well copies=2, with deduplication enabled.
Click to expand...
If you have the possibility of having two separate devices, I'd favor a mirror instead of using "copies=2" on a single device.


Deduplication on (Open) ZFS​

  1. Dedup Performance by Matt Ahrens at OpenZFS Developer Summit 2017 (see: "A proposal for 1,000x better dedup performance")
  2. Introducing Fast Dedup by Allan Jude at OpenZFS Developer Summit 2023 (see: "Introducing Fast Dedup")
    What makes [the current] dedup slow, and more.
Personally, I'd think twice about enabling dedup in its currrent form; on a single device I wouldn't even consider it. Appetiser from Matthew Ahrens [2017, from #1]:
Dedup performance sucks...
Click to expand...
Dan Langille in ZFS for Newbies at ca. 4:30 min - 2019*:
Don't use dedup. Friends don't let friends use dedup. You'll hear about it and you'll wanna use it, but don't do it. Use compression instead.
Click to expand...

I'd wait patiently for fast dedup to land in FreeBSD and make an assessment based on its characteristics.

Fast Dedup* - released in OpenZFS 2.3.0 - Jan 14, 2025​

  1. Introducing Fast Dedup by Allan Jude at OpenZFS Developer Summit 2023 (see: "Introducing Fast Dedup")
  2. Fast Dedup #15896 - Fast Dedup Review Guide
    Ongoing monitoring of development of Fast Dedup on OpenZFS' github.
  3. Introducing OpenZFS Fast Dedup by Jim Salter, November 6, 2024.
  4. Applying the ARC Algorithm to the ARC by Allan Jude, November 13, 2024.
    How Does ZFS Dedup Work? and more ...
  5. Fast Dedup with ZFS: Smarter Storage for Modern Workloads by Allan Jude and Kris Moore, 18 June, 2025.

zfs-2.3.0 will be the first release with Fast Dedup, currently at zfs-2.3.0-rc3 prerelease; follow OpenZFS releases. -> It has arrived, waiting to land officialy in a FreeBSD release.

___
P.S.
Ferocious1881 said:
[...] And looks like is all of it, after reading https://docs.oracle.com/cd/E36784_01/html/E36835/gazsd.html#scrolltoc
Click to expand...
You seem to refer to / rely often on Oracle documentation; please note that (Open)ZFS in FreeBSD is not Oracle ZFS; from Wikipedia-ZFS:
According to Matt Ahrens, one of the main architects of ZFS, over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions as of 2019, making “Oracle ZFS” and “OpenZFS” politically and technologically incompatible.
Click to expand...
I suggest consulting the FreeBSD manpages, like zfs-rename(8) or the relevant ZFS man pages for your particular FreeBSD version at Man Pages - OpenZFS, like zfsprops(7) - dedup

* Edit: updated;
2025-07-12 updated and added youtube, #5 of the list above.
 
Last edited:
Well, OP should just let the installer go with defaults for ZFS on root, and do all the tweaking later, after installation. With ZFS, that's possible, and that's what got me sold on ZFS in the first place. ?
 
astyle said:
Well, OP should just let the installer go with defaults for ZFS on root, and do all the tweaking later, after installation. With ZFS, that's possible, and that's what got me sold on ZFS in the first place. ?
Click to expand...
Installer also allows encrypt hard drive ( i believe root, also swap ? and mirror it if i recall it correctly) and adding user can also encrypt Home partition ( i think )
 
GlitchyDot said:
Installer also allows encrypt hard drive ( i believe root, also swap ? and mirror it if i recall it correctly) and adding user can also encrypt Home partition ( i think )
Click to expand...
Yep, even with ZFS. And ZFS allows you to do all that and more later, any time after installation. Off the top of my head, I don't know of other filesystems that allow you to do all that at your convenience.

With ZFS, you can forget about partitioning altogether, and adjust size limits of /home and other datasets any time.
 
astyle said:
Yep, even with ZFS. And ZFS allows you to do all that and more later, any time after installation. Off the top of my head, I don't know of other filesystems that allow you to do all that at your convenience.

With ZFS, you can forget about partitioning altogether, and adjust size limits of /home and other datasets any time.
Click to expand...
Some things cannot be done later, like changing the utf8only property.
 
You must log in or register to reply here.
Back
Top