Solved 13.1 breaks my encrypted disk setup

A

arturb

[Updated to include results of my newest tests]

Prior to version 13.1, I had a working disk setup that I've been using to
install FreeBSD. This has changed recently and I can no longer use my custom
partitioning layout with encryption due to an error.

I am in the process of troubleshooting it, but the boot process is evolving and
I think it's good opportunity to establish a working and correct partitioning
procedure for an encrypted system.

Let me describe and discuss the issue.

I based my setup on a number of forum posts or blogs and videos. It was some
years ago. The aim was to have:
1) BIOS [+optional EFI] support
2) Encrypted system partition including boot environments
3) UFS filesystem.

This is what my setup script is told to do in order to achieve that:

1) Create a new GPT partition table
/bin/dd if=/dev/zero of=/dev/ada0 bs=512 count=1
/sbin/gpart destroy -F ada0
/sbin/gpart create -s gpt ada0
2) [optional] Add an EFI boot partition [ada0p1]
gpart add -t efi -s 260M -l efiboot ada0
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /mnt

mkdir -p /mnt/efi/boot
cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh

umount /dev/ada0p1
3) Add a BIOS boot (pmbr) partition [ada0p2 or ada0p1 if no EFI]
gpart add -t freebsd-boot -s 512K -l biosboot ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 2 ada0
4) Add system partition and encrypt it [ada0p3]
gpart add -t freebsd-ufs -l cryptsys ada0
geli init -b -e aes-xts -l 256 -s 4096 /dev/ada0p3
geli attach /dev/ada0p3
geli configure -g /dev/ada0p3
4.1) Add internal partition table and partitions [ada0p3.eli]
gpart create -s bsd /dev/ada0p3.eli
# / | ada0p3.elia
# swap | ada0p3.elib
# /var | ada0p3.elid
# /tmp | ada0p3.elie
# /usr | ada0p3.elif
gpart add -t freebsd-ufs -a 4K -s 3G /dev/ada0p3.eli
gpart add -t freebsd-swap -a 4K -s 4G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 10G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K -s 1G /dev/ada0p3.eli
gpart add -t freebsd-ufs -a 4K /dev/ada0p3.eli

newfs -U /dev/ada0p3.elia
newfs -U /dev/ada0p3.elid
newfs -U /dev/ada0p3.elie
newfs -U /dev/ada0p3.elif
5) Mount system partitions
mount /dev/ada0p3.elia /mnt
mkdir -p /mnt/var /mnt/tmp /mnt/usr
mount /dev/ada0p3.elid /mnt/var
mount /dev/ada0p3.elie /mnt/tmp
mount /dev/ada0p3.elif /mnt/usr
6) Configure /boot/loader.conf and /etc/fstab
printf 'geom_eli_load="YES"\n' > /tmp/bsdinstall_boot/loader.conf
printf 'vfs.root.mountfrom="ufs:ada0p3.elia"\n' >> /tmp/bsdinstall_boot/loader.conf

printf '<as below>' >> /tmp/bsdinstall_etc/fstab
/dev/ada0p3.elia / ufs rw 1 1
/dev/ada0p3.elib none swap sw,late 0 0
/dev/ada0p3.elid /var ufs rw 0 2
/dev/ada0p3.elie /tmp ufs rw 0 2
/dev/ada0p3.elif /usr ufs rw 0 2

This is what I consider a "robust" setup. It doesn't work with 13.1-RELEASE.

Some questions:
1) The first matter
ad. 2) [optional] Add an EFI boot partition [ada0p1]

What is the required size for the EFI partition? I remember having to set it to 260M in my tests, as anything below was considered too low. This was enforced by some online resources I read for comparison.
The source [1], however, lists a following example:

+# gpart add -a 4K -t efi -s 200M ada0
vs what I use:
-# gpart add -t efi -s 260M ada0

I am not sure where the difference comes from.
2) Second matter
ad. 2) [optional] Add an EFI boot partition [ada0p1]

The middle section could perhaps be modernised, as per [1] again,
but I'm not certain it works with legacy boot (BIOS); the article says
'UEFI/GPT/MBR/UFS/ZFS (13.0 and later)', where the loader.efi takes
care of stages 1-3 in contrast to boot1.efi.

-# mkdir -p /mnt/efi/boot
-# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
-# printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh

+# mkdir -p /mnt/efi/boot /mnt/efi/freebsd
+# cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
+# cp /boot/loader.efi /mnt/efi/freebsd/

So should boot1.efi be copied? Could loader.efi alone suffice? Is startup.nsh needed? Testing it on a device with Legacy BIOS disabled to see if these EFI settings work correctly is in my backlog, but perhaps someone can tell already.
3) Third matter - the error at boot
I updated my setup from using a separate boot partition at the start of the encrypted system drive and mounted at /bootfs in /etc/fstab (with a symbolic link for /boot@ -> bootfs/boot) because that was included in the guides I read when I first started using FreeBSD and had no idea whatsoever about the boot process.
I don't know why that was, but it seems to me a needless complication for which I have no use. It worked however and now it stopped working with the following error.

[after GELI decryption]
FreeBSD/x86 bootstrap loader, Revision 1.1
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
Click to expand...

My updated setup which does not create a separate partition for /boot, and which I outlined above, ends in the exact same error. It leads me to believe that both setups (separate boot pool partition vs boot at root partition) are equally valid, but something in 13.1-RELEASE breaks them.

All other attempts at tinkering with partition layouts, setting flags, etc. end up in:

[after GELI decryption]
config: not a directory.
loader: not a directory.
gptboot: No /boot/loader on 0:ad(0p3)
kernel: not a directory.
gptboot: No /boot/kernel/kernel on 0:ad(0p3)

FreeBSD/x86 boot
Default: 0:ad(0p3)/boot/kernel/kernel
boot:
Click to expand...

This seems more serious than the previous error. Therefore I am focusing on finding out what is going on with that LUA loader file. I reported it previously [here] and [here].

My primary aim is to establish a working solution, but more generally it would
be useful to see if the changes made to the boot process in recent time can be
integrated to modernise this setup (as per my EFI partition annotations above).

This may be a niche use-case in a free software project, but the FreeBSD
installer, or documentation at least, could really be updated to incorporate UFS
encryption and the ongoing changes to the boot process. To my knowledge, there
is no official guide on how a UFS system with encrypted boot environments can be
prepared and the user is left to drudging through old blog and forum posts that
may or may not be viable anymore. To that end, I think it would be useful to
discuss this here and elsewhere and perhaps summarise it all into a guide or an
entry in the Handbook. I've spent many hours reading about and testing the boot
process and I think some of it should be spared for other users.

Finally, a word of gratitude to the community members such as Robonuggie, Allan
Jude, Klara and all the bloggers and forum users who make the difficult slightly
less so.

Sources:
[1] - Klara - The FreeBSD Boot Process

[edited to fix some mistakes and improve readability]
 
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
Click to expand...

Maybe FreeBSD bug 264282 – BIOS boot from GELI encrypted broken / 'currdev' set to wrong string.

… I've bisect this in 13.1-RELEASE. It was broken in commit bc9154a208248, which was cherry picked from b4cb3fe0e39a. …
Click to expand...

github.com

loader: implement mount/unmount rootfs · freebsd/freebsd-src@bc9154a

We want to keep our root file system open to preserve bcache segment between file accesses, thus reducing physical disk IO. Reviewed by: imp, allanjude, kevans (previous version) Differential Revi...
github.com github.com
 
grahamperrin said:
Maybe FreeBSD bug 264282 – BIOS boot from GELI encrypted broken / 'currdev' set to wrong string.



github.com

loader: implement mount/unmount rootfs · freebsd/freebsd-src@bc9154a

We want to keep our root file system open to preserve bcache segment between file accesses, thus reducing physical disk IO. Reviewed by: imp, allanjude, kevans (previous version) Differential Revi...
github.com github.com
Click to expand...
Indeed looks like it. I did some more testing based on historical reports of this issue (like getting away with the encrypted partition BSD scheme and using raw encrypted / and the problem persists there. At the same time, simply swapping /boot/loader to the one from 13.0 lets one boot correctly.

Thanks for the link. I will bump it with my info.
 
arturb said:
geli init -b -e aes-xts -l 256 -s 4096 /dev/ada0p3
...
geli configure -g /dev/ada0p3
Click to expand...
This can be shortened to
Code: 
geli init -bg -l 256 -s 4096 ada0p3
Encryption algorithm AES-XTS is the default, no need to set it explicitly.

arturb said:
What is the required size for the EFI partition? I remember having to set it to 260M in my tests, as anything below was considered too low.
Click to expand...
Considered to low by whom?

Assuming this is a FreeBSD only installation (not multi OS), then the minimum size of the ESP is determined by the size of loader.efi(8):
Code: 
ls -lh /boot/loader.efi
-r-xr-xr-x  2 root  wheel   875K Jan 18 15:50 /boot/loader.efi
A ESP with a size of 1m would already suffice to fit the loader.

arturb said:
So should boot1.efi be copied? Could loader.efi alone suffice?
Click to expand...
loader.efi(8) alone will suffice.

boot1.efi(8)
Code: 
DESCRIPTION
     boot1.efi has been deprecated and will be removed from a future release.
     loader.efi(8) handles all its former use cases with more flexibility.

     On UEFI systems, boot1.efi loads /boot/loader.efi from the default root
     file system and transfers execution there.

arturb said:
Is startup.nsh needed?
Click to expand...
The menu guided installation doesn't drop one in the ESP, it's save to say it's not needed.

EDIT:
I saw your posting on freebsd-questions, instead of using bsdinstall(8), write your personal installation script and execute that script.

bsdinstall(8) is not necessary to create a custom installation script. All utilities it uses are available from system to install manually or scripted from any file.
 
Last edited:
T-Daemon said:
Encryption algorithm AES-XTS is the default, no need to set it explicitly.
Click to expand...
I sometimes like to have things set explicitly to have important options included in the code.

T-Daemon said:
Considered to low by whom?
Click to expand...
To be quite honest, I can't exactly remember, but I think it might have been that on one of my installs I chose some smaller EFI size and then couldn't boot. A case of post hoc ergo propter hoc, most likely. It is now clarified.

T-Daemon said:
loader.efi(8) alone will suffice.
Click to expand...
It does, as also mentioned FreeBSD 13.0 release notes under 'Boot Loader Changes'.
Importantly, that this partition ought to be mounted to /boot/efi as msdosfs, so it needs a line in /etc/fstab.

T-Daemon said:
The menu guided installation doesn't drop one in the ESP, it's save to say it's not needed.
Click to expand...
You're correct again and confirm my own test.

T-Daemon said:
I saw your posting on freebsd-questions, instead of using bsdinstall(8), write your personal installation script and execute that script.

bsdinstall(8) is not necessary to create a custom installation script. All utilities it uses are available from system to install manually or scripted from any file.
Click to expand...
Still, bsdinstall(8) is useful because it lets you skip some steps, particularly checksum checks and untaring the archives properly.
I will consider skipping bsdinstall(8) once I learn more about the install process - right now, even using the program, I am still struggling to produce a fully functional install script that does everything that I want.

Thank you for correcting and confirming many of my doubts.
 
That's just the result of a dirty filesystem. It's not because the efi partition was mounted, it was because it wasn't unmounted properly (this left it in a 'dirty' state)[*]. Although you could certainly make the case that a mounted filesystem can be handled incorrectly and a filesystem that's not mounted at all could never get in a dirty state. It's still not the mounting itself that's problematic.

[*] In this case I suspect the system was rebooted while something was still writing to the disk. That caused the filesystem to get corrupted (FAT is notoriously awful in this respect).
 
Anyway, I don't have efi mounted all the time, no point, I don't have to constantly edit files there. I do have it in fstab with the noauto flag, so I can easily mount it whenever I need to (after updates for example, to update the loader).
 
SirDice said:
Anyway, I don't have efi mounted all the time, no point, I don't have to constantly edit files there. I do have it in fstab with the noauto flag, so I can easily mount it whenever I need to (after updates for example, to update the loader).
Click to expand...
I like to manually mount partitions and chroot. when you lift a system, something may not be working correctly, it's often my case, I work with kernel.
 
<https://media.discordapp.net/attach...980569872759156746/PXL_20220529_203032524.jpg>

…
Click to expand...

1653933053186.png


fsck -y -T ffs:-R -T ufs:-R seems insane for msdosfos.

fsck(8)
 
arturb said:
1) Create a new GPT partition table
/bin/dd if=/dev/zero of=/dev/ada0 bs=512 count=1
/sbin/gpart destroy -F ada0
/sbin/gpart create -s gpt ada0 2) [optional] Add an EFI boot partition [ada0p1]
gpart add -t efi -s 260M -l efiboot ada0
newfs_msdos /dev/ada0p1
mount -t msdosfs /dev/ada0p1 /mnt

mkdir -p /mnt/efi/boot
cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi
printf 'bootx64.efi' > /mnt/efi/boot/startup.nsh

umount /dev/ada0p1 3) Add a BIOS boot (pmbr) partition [ada0p2 or ada0p1 if no EFI]
gpart add -t freebsd-boot -s 512K -l biosboot ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 2 ada0
Click to expand...
why don't you create a script instead of having to type this whole command, and a waste in my opinion. is simpler, script in sh, you automate this whole process, if it is your case
 
Dimitri Chuikov said:
why don't you create a script instead of having to type this whole command, and a waste in my opinion. is simpler, script in sh, you automate this whole process, if it is your case
Click to expand...
Thank you. I do in fact have a script and it's that script that's wasting its time while I drink tea and read mailing lists.

grahamperrin said:
View attachment 14089

fsck -y -T ffs:-R -T ufs:-R.
Click to expand...
Reinstalling the system might be faster than typing this in.


EDIT: I'm going to mark this as solved. The encryption bug was diagnosed and traced by yamagi on Bugzilla, as referenced by grahamperrin in an earlier post here. Thanks everyone.
 
Last edited:
You must log in or register to reply here.
Back
Top