[Updated to include results of my newest tests]
Prior to version 13.1, I had a working disk setup that I've been using to
install FreeBSD. This has changed recently and I can no longer use my custom
partitioning layout with encryption due to an error.
I am in the process of troubleshooting it, but the boot process is evolving and
I think it's good opportunity to establish a working and correct partitioning
procedure for an encrypted system.
Let me describe and discuss the issue.
I based my setup on a number of forum posts or blogs and videos. It was some
years ago. The aim was to have:
1) BIOS [+optional EFI] support
2) Encrypted system partition including boot environments
3) UFS filesystem.
This is what my setup script is told to do in order to achieve that:
1) Create a new GPT partition table
2) [optional] Add an EFI boot partition [ada0p1]
3) Add a BIOS boot (pmbr) partition [ada0p2 or ada0p1 if no EFI]
4) Add system partition and encrypt it [ada0p3]
4.1) Add internal partition table and partitions [ada0p3.eli]
5) Mount system partitions
6) Configure /boot/loader.conf and /etc/fstab
This is what I consider a "robust" setup. It doesn't work with 13.1-RELEASE.
Some questions:
1) The first matter
2) Second matter
3) Third matter - the error at boot
My primary aim is to establish a working solution, but more generally it would
be useful to see if the changes made to the boot process in recent time can be
integrated to modernise this setup (as per my EFI partition annotations above).
This may be a niche use-case in a free software project, but the FreeBSD
installer, or documentation at least, could really be updated to incorporate UFS
encryption and the ongoing changes to the boot process. To my knowledge, there
is no official guide on how a UFS system with encrypted boot environments can be
prepared and the user is left to drudging through old blog and forum posts that
may or may not be viable anymore. To that end, I think it would be useful to
discuss this here and elsewhere and perhaps summarise it all into a guide or an
entry in the Handbook. I've spent many hours reading about and testing the boot
process and I think some of it should be spared for other users.
Finally, a word of gratitude to the community members such as Robonuggie, Allan
Jude, Klara and all the bloggers and forum users who make the difficult slightly
less so.
Sources:
[1] - Klara - The FreeBSD Boot Process
[edited to fix some mistakes and improve readability]
Prior to version 13.1, I had a working disk setup that I've been using to
install FreeBSD. This has changed recently and I can no longer use my custom
partitioning layout with encryption due to an error.
I am in the process of troubleshooting it, but the boot process is evolving and
I think it's good opportunity to establish a working and correct partitioning
procedure for an encrypted system.
Let me describe and discuss the issue.
I based my setup on a number of forum posts or blogs and videos. It was some
years ago. The aim was to have:
1) BIOS [+optional EFI] support
2) Encrypted system partition including boot environments
3) UFS filesystem.
This is what my setup script is told to do in order to achieve that:
1) Create a new GPT partition table
/bin/dd if=/dev/zero of=/dev/ada0 bs=512 count=1
/sbin/gpart destroy -F ada0
/sbin/gpart create -s gpt ada0
/sbin/gpart destroy -F ada0
/sbin/gpart create -s gpt ada0
gpart add -t efi -s 260M -l efiboot ada0
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /mnt
mkdir -p /mnt/efi/boot
cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh
umount /dev/ada0p1
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /mnt
mkdir -p /mnt/efi/boot
cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh
umount /dev/ada0p1
gpart add -t freebsd-boot -s 512K -l biosboot ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 2 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 2 ada0
gpart add -t freebsd-ufs -l cryptsys ada0
geli init -b -e aes-xts -l 256 -s 4096 /dev/ada0p3
geli attach /dev/ada0p3
geli configure -g /dev/ada0p3
geli init -b -e aes-xts -l 256 -s 4096 /dev/ada0p3
geli attach /dev/ada0p3
geli configure -g /dev/ada0p3
gpart create -s bsd /dev/ada0p3.eli
# / | ada0p3.elia
# swap | ada0p3.elib
# /var | ada0p3.elid
# /tmp | ada0p3.elie
# /usr | ada0p3.elif
gpart add -t freebsd-ufs -a 4K -s 3G /dev/ada0p3.eli
gpart add -t freebsd-swap -a 4K -s 4G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 10G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 1G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K /dev/ada0p3.eli
newfs -U /dev/ada0p3.elia
newfs -U /dev/ada0p3.elid
newfs -U /dev/ada0p3.elie
newfs -U /dev/ada0p3.elif
# / | ada0p3.elia
# swap | ada0p3.elib
# /var | ada0p3.elid
# /tmp | ada0p3.elie
# /usr | ada0p3.elif
gpart add -t freebsd-ufs -a 4K -s 3G /dev/ada0p3.eli
gpart add -t freebsd-swap -a 4K -s 4G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 10G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 1G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K /dev/ada0p3.eli
newfs -U /dev/ada0p3.elia
newfs -U /dev/ada0p3.elid
newfs -U /dev/ada0p3.elie
newfs -U /dev/ada0p3.elif
mount /dev/ada0p3.elia /mnt
mkdir -p /mnt/var /mnt/tmp /mnt/usr
mount /dev/ada0p3.elid /mnt/var
mount /dev/ada0p3.elie /mnt/tmp
mount /dev/ada0p3.elif /mnt/usr
mkdir -p /mnt/var /mnt/tmp /mnt/usr
mount /dev/ada0p3.elid /mnt/var
mount /dev/ada0p3.elie /mnt/tmp
mount /dev/ada0p3.elif /mnt/usr
printf 'geom_eli_load="YES"\n' > /tmp/bsdinstall_boot/loader.conf
printf 'vfs.root.mountfrom="ufs:ada0p3.elia"\n' >> /tmp/bsdinstall_boot/loader.conf
printf '<as below>' >> /tmp/bsdinstall_etc/fstab
/dev/ada0p3.elia / ufs rw 1 1
/dev/ada0p3.elib none swap sw,late 0 0
/dev/ada0p3.elid /var ufs rw 0 2
/dev/ada0p3.elie /tmp ufs rw 0 2
/dev/ada0p3.elif /usr ufs rw 0 2
printf 'vfs.root.mountfrom="ufs:ada0p3.elia"\n' >> /tmp/bsdinstall_boot/loader.conf
printf '<as below>' >> /tmp/bsdinstall_etc/fstab
/dev/ada0p3.elia / ufs rw 1 1
/dev/ada0p3.elib none swap sw,late 0 0
/dev/ada0p3.elid /var ufs rw 0 2
/dev/ada0p3.elie /tmp ufs rw 0 2
/dev/ada0p3.elif /usr ufs rw 0 2
This is what I consider a "robust" setup. It doesn't work with 13.1-RELEASE.
Some questions:
1) The first matter
ad. 2) [optional] Add an EFI boot partition [ada0p1]
What is the required size for the EFI partition? I remember having to set it to 260M in my tests, as anything below was considered too low. This was enforced by some online resources I read for comparison.
The source [1], however, lists a following example:
+# gpart add -a 4K -t efi -s 200M ada0
vs what I use:
-# gpart add -t efi -s 260M ada0
I am not sure where the difference comes from.
What is the required size for the EFI partition? I remember having to set it to 260M in my tests, as anything below was considered too low. This was enforced by some online resources I read for comparison.
The source [1], however, lists a following example:
+# gpart add -a 4K -t efi -s 200M ada0
vs what I use:
-# gpart add -t efi -s 260M ada0
I am not sure where the difference comes from.
ad. 2) [optional] Add an EFI boot partition [ada0p1]
The middle section could perhaps be modernised, as per [1] again,
but I'm not certain it works with legacy boot (BIOS); the article says
'UEFI/GPT/MBR/UFS/ZFS (13.0 and later)', where the loader.efi takes
care of stages 1-3 in contrast to boot1.efi.
-# mkdir -p /mnt/efi/boot
-# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
-# printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh
+# mkdir -p /mnt/efi/boot /mnt/efi/freebsd
+# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
+# cp /boot/loader.efi /mnt/efi/freebsd/
So should boot1.efi be copied? Could loader.efi alone suffice? Is startup.nsh needed? Testing it on a device with Legacy BIOS disabled to see if these EFI settings work correctly is in my backlog, but perhaps someone can tell already.
The middle section could perhaps be modernised, as per [1] again,
but I'm not certain it works with legacy boot (BIOS); the article says
'UEFI/GPT/MBR/UFS/ZFS (13.0 and later)', where the loader.efi takes
care of stages 1-3 in contrast to boot1.efi.
-# mkdir -p /mnt/efi/boot
-# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
-# printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh
+# mkdir -p /mnt/efi/boot /mnt/efi/freebsd
+# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
+# cp /boot/loader.efi /mnt/efi/freebsd/
So should boot1.efi be copied? Could loader.efi alone suffice? Is startup.nsh needed? Testing it on a device with Legacy BIOS disabled to see if these EFI settings work correctly is in my backlog, but perhaps someone can tell already.
I updated my setup from using a separate boot partition at the start of the encrypted system drive and mounted at /bootfs in /etc/fstab (with a symbolic link for /boot@ -> bootfs/boot) because that was included in the guides I read when I first started using FreeBSD and had no idea whatsoever about the boot process.
I don't know why that was, but it seems to me a needless complication for which I have no use. It worked however and now it stopped working with the following error.
[after GELI decryption]
My updated setup which does not create a separate partition for /boot, and which I outlined above, ends in the exact same error. It leads me to believe that both setups (separate boot pool partition vs boot at root partition) are equally valid, but something in 13.1-RELEASE breaks them.
All other attempts at tinkering with partition layouts, setting flags, etc. end up in:
[after GELI decryption]
This seems more serious than the previous error. Therefore I am focusing on finding out what is going on with that LUA loader file. I reported it previously [here] and [here].
I don't know why that was, but it seems to me a needless complication for which I have no use. It worked however and now it stopped working with the following error.
[after GELI decryption]
FreeBSD/x86 bootstrap loader, Revision 1.1
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
My updated setup which does not create a separate partition for /boot, and which I outlined above, ends in the exact same error. It leads me to believe that both setups (separate boot pool partition vs boot at root partition) are equally valid, but something in 13.1-RELEASE breaks them.
All other attempts at tinkering with partition layouts, setting flags, etc. end up in:
[after GELI decryption]
config: not a directory.
loader: not a directory.
gptboot: No /boot/loader on 0:ad(0p3)
kernel: not a directory.
gptboot: No /boot/kernel/kernel on 0:ad(0p3)
FreeBSD/x86 boot
Default: 0:ad(0p3)/boot/kernel/kernel
boot:
This seems more serious than the previous error. Therefore I am focusing on finding out what is going on with that LUA loader file. I reported it previously [here] and [here].
My primary aim is to establish a working solution, but more generally it would
be useful to see if the changes made to the boot process in recent time can be
integrated to modernise this setup (as per my EFI partition annotations above).
This may be a niche use-case in a free software project, but the FreeBSD
installer, or documentation at least, could really be updated to incorporate UFS
encryption and the ongoing changes to the boot process. To my knowledge, there
is no official guide on how a UFS system with encrypted boot environments can be
prepared and the user is left to drudging through old blog and forum posts that
may or may not be viable anymore. To that end, I think it would be useful to
discuss this here and elsewhere and perhaps summarise it all into a guide or an
entry in the Handbook. I've spent many hours reading about and testing the boot
process and I think some of it should be spared for other users.
Finally, a word of gratitude to the community members such as Robonuggie, Allan
Jude, Klara and all the bloggers and forum users who make the difficult slightly
less so.
Sources:
[1] - Klara - The FreeBSD Boot Process
[edited to fix some mistakes and improve readability]