I read about the topic a lot in the last few days. What I found is intersting with my limited Unix knowledge. It turned out I need at least the kernel and some sort of in-memory root like initrd to be able to boot properly, so those cannot be encrypted. I need to support the actual encryption ofc. and I need to get the keys somehow from the admin. All of these unencrypted parts must be packed into an EFI file, that the UEFI can verify with secure boot and load after that.
It is relative hard to rebuild that EFI file after every kernel upgrade. It is possible to split it up with a boot loader that supports signature verification before loading something. In that case the boot loader and its configuration must be signed. With GRUB and Linux it is possible to find such solutions.
https://ruderich.org/simon/notes/secure-boot-with-grub-and-signed-linux-and-initrd Another option is having a different kernel just for decrypting the system. In that case we don't need to modify the EFI file by kernel upgrade, but it is a challange to give the control to the second kernel. Linux ppl use kexec for that, but even with that one needs to give the encryption keys to the second kernel somehow and solutions I found for that are pretty much insecure, for example:
https://github.com/flowztul/keyexec writes the keys to the disk as far as I can tell and even if it removes them after that, they can be stolen with tools designed for restoring removed files. Afaik FreeBSD has something like kload, which should be similar to kexec, but I haven't had the time to read about it. I guess even with that I won't be able to solve the problem this way, so better to stick with the first solution, which is having a boot loader that can verify signatures and updating the kernel file and its signature after every kernel upgrade. That can be done by a hook automagically, but I need to store the private keys on the encrypted partition, which is slightly less secure then signing the boot loader only once and store the private key for signing in a pendrive which I never mount again.
I think these solutions are effective even against the "evil maid attack"
https://en.wikipedia.org/wiki/Evil_maid_attack where the attacker not necessarily steals the hardware, but adds some unwanted software to it to get the keys. Even with this protection it is possible to get the keys as simply as hacking the keyboard. With software I think it is very hard to attack this solution. We could try to inject our code between the EFI file and the encrypted system if it is possible to force the boot loader to load that instead of the system, but it is unlikely that this can be done.
We might want to add remote unlocking so for example we could wake up and unlock the server after a power shortage even from a different country if we move the VPN server to the router. This is a very useful feature, but it opens up new ways to attack our system, mostly maid in the middle (MITM) like attacks. =] It is typical in Linux systems to add some minimalist SSH server like dropbear to the boot loader, but the private keys of that server must be unencrypted, so our maid can esily get them. After that she can try to do ARP poisoning for example. We can detect that by watching ARP communication:
http://blog.sevagas.com/?MITM-arp-spoofing I guess there are many other ways to do MITM on such a system and I guess it is impossible to protect against all of them. There are other ways to access our remote system. Many server boards contain a thing called BMC. It is some sort of small computer built in the server to monitor it. As far as I know it has a dedicated ethernet port and can be accessible through IPMI possibly on a different network. Unfortunately my home server is built on a workstation motherboard, because it had half the price compared to server boards and I heard that IPMI implementations have a lot of vulnerabilities, so I decided that I won't need that feature. Some people claim that there is another way to send the keys something called serial network interface (SNI). I did not find much about that. As far as I understand it is converting network communication to serial one. I guess it is better because we won't need network drivers for it, and serial communication is supported by the kernel. I would skip this type of very low level solutions, because they need a lot of expertise I guess and as far as I can tell having network communication in the boot loader is no longer an issue in most of the cases.
So if we need remote unlocking, then there is a good chance our evil maid will steal our encryption keys along with the data and we might not even notice it. For me the remote unlocking is currently more important than protecting my system against evil maids. I just want to protect it against attackers who steal the drives and try to decrypt the data. Which means that they have access to the hardware only a single time, I notice that they stole my disks and they don't have the keys to decrypt the data on them. Protecting against such attacks does not require full disk encryption. All I need is encrypting the data and the swap. Though I tried to find a solution that does full disk encryption with split up EFI file and detached signatures, something similar I already linked about GRUB+ LUKS
https://ruderich.org/simon/notes/secure-boot-with-grub-and-signed-linux-and-initrd , but I had no luck. According to the handbook geli supports full disk encryption
https://www.freebsd.org/doc/handbook/disks-encrypting.html , but according to the forum it doesn't.
https://forums.freebsd.org/threads/bring-up-networking-earlier-during-boot.49227/ Maybe that forum post is outdated, I don't know. I'll read the manual of geli later to find out, but even if it is possible, I'll need a boot loader that can verify signatures, can get the keys through SSH and supports geli and ZFS. While it is possible to have such a boot loader I think it is unlikely there is one, because it is not advertised. Google cannot find it or I do something totally wrong.
I asked about ZFS native encryption too. As I thought with that ZFS can verify encrypted blocks. So unlike geli (or LUKS in the case of Linux) it does not have to decrypt data to be able to verify signatures and fix the corrupted data if there is redundancy. From performance perspective this might not matter that much. I don't know. It looks like ZFS native encryption supports comes only with FreeBSD 13 and based on the release history I would expect that in late 2021 or early 2022. So ZFS native encryption would be nice, but it is impossible currently in FreeBSD. Even with GRUB and Linux it would not be that stable, there are still unsolved issues in the git repos if I check. So we can revive this perfect full disk encryption topic in 2022.
For now I think I'll do what you suggested, which means encrypting the swap and data with geli and leave the rest of the system unencrypted. Some people claimed that the best way to do this with data is having an encrypted jail partition and use ezjail.
https://forums.freebsd.org/threads/bring-up-networking-earlier-during-boot.49227/ I am not entirely convinced about that, but I'll check. I don't think I'll do secure boot either, it makes no sense with an unencrypted system unless I sign and verify every single file. Without built in support I don't think I will do it. I think this is sufficient for my goals and in practice there is no better solution I know of if I want to stick with FreeBSD. I think I want to, I tried to communicate with Alpine developers a few years ago, but people here are a lot friendlier. In my experience this is generally true for all Linux, I have no idea why.