Solved Using GRUB2 to chainload GELI-encrypted root on ZFS FreeBSD 10.1-RC2

This post was originally titled "Using GRUB2 to chainload FreeBSD from protective MBR (PMBR)".

I have Windows 8, Arch Linux, and FreeBSD 10.0 all installed on the same disk. All my partitions are in the GPT format (not MBR). I'm able to boot into Windows and Linux, but not FreeBSD.

For FreeBSD, I have 4 partitions: freebsd-boot, freebsd-zfs for /boot, freebsd-swap, freebsd-zfs for /. I installed the bootloader code like so: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i <index_of_freebsd-boot_partition> <device>.

I used dd if=/dev/sda of=/boot/sda.mbr bs=512 count=1 to backup my MBR to my EFI System Partition (ESP), which is mounted at /boot.

When I use file -k /boot/sda.mbr, I see
Code:
DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,1), end-CHS (0x3ff,254,63), startsector 1, 1953525167 sectors

My /etc/grub.d/40_custom on Arch Linux has the following menu entry for FreeBSD, which I added manually:
Code:
menuentry 'FreeBSD 10.0' {
    set root='hd0,gpt11'
    chainloader (hd0,gpt2)/sda.mbr
}

All the partitions on my disk are using GPT. gpt11 is my FreeBSD root partition, which uses ZFS. gpt2 is my ESP. If I try booting the FreeBSD entry, I see the following GRUB2 error:
Code:
error: cannot load image.

If I look at the /boot/sda.mbr file, the strings that I can make out are "EFI" and "PART". I also see "Boot loader too large", "Invalid partition table", "I/O error loading boot loader", and "Missing boot loader". I'm not sure if those are just error strings that are defined for the stage 1 (or 0 or whatever) boot loader, or if those strings indicate that there's something wrong with my MBR.

The reason why I'm trying to chainload my MBR is because I'm unable to boot FreeBSD at all (boot managers don't even list it). I've even installed rEFInd, which is unable to detect FreeBSD on my disk. Since I'm using FreeBSD 10.0, ESP is unsupported, so I cannot boot from ESP. I thought of just waiting for FreeBSD 10.1, which does have ESP support; however, 10.1 won't support ESP where root is on ZFS.

My system has Secure Boot, Quiet Boot, and Fast Startup all disabled. I have CSM enabled, which allows me to boot MBR-based disks. I've booted plenty of MBR-based LiveCDs and LiveUSBs on my machine.

At this point, I suspect two things: either (1) my MBR is corrupt; or (2) my UEFI firmware is unable to boot from MBR when it detects that I'm using GPT. I'm trying to chainload to get around number 2; however, GRUB2 cannot load the MBR image, which makes me suspect that there's a problem with my MBR.

I've already tried using the following GRUB2 options when booting FreeBSD:
Code:
set pager=1
set debug=all

The output was not helpful.

Any ideas? I'm stuck on where to go to from here. Is what I'm trying even possible? Maybe my UEFI firmware and GRUB2 and rEFInd see that my MBR is a protective MBR (PMBR) and that I have a GPT-based disk, so they ignore whatever else is in the MBR because they expect to boot GPT from ESP. I also wonder if having Windows, Linux, and FreeBSD all on the same disk is somehow overloading the MBR.

EDIT: The solution is posted in message #21: https://forums.freebsd.org/threads/...d-from-protective-mbr-pmbr.48442/#post-271242.
 
Last edited:
Re: Using GRUB2 to chainload FreeBSD from protective MBR (PM

Rather than chainloading the FreeBSD bootcode, try just loading the loader:
Code:
menuentry 'FreeBSD 10.0' {
    set root='hd0,gpt11'
    kfreebsd /boot/loader
}

Also, the size of the partitions is not shown. Remember that freebsd-boot partitions should be no larger than 512K (yes, K).
 
The partitions on the disk:
Code:
/dev/sda1     400M      NTFS
/dev/sda2     300M      VFAT
/dev/sda3     128M      Microsoft reserved
/dev/sda4     99.7G     NTFS
/dev/sda5     350M      NTFS
/dev/sda6     400G      ext4
/dev/sda7     17G       NTFS
/dev/sda8     512K      freebsd-boot
/dev/sda9     1G        freebsd-zfs: /boot
/dev/sda10    4G        freebsd-swap
/dev/sda11    408.7G    freebsd-zfs: /

I've tried many variants of the kfreebsd line:
Code:
kfreebsd /boot/kernel/kernel
kfreebsd /boot/zfsloader
kfreebsd /boot/loader
kfreebsd /freebsd@/boot/kernel/kernel
kfreebsd /freebsd@/boot/zfsloader
kfreebsd /freebsd@/boot/loader
kfreebsd /@/boot/kernel/kernel
kfreebsd /@/boot/zfsloader
kfreebsd /@/boot/loader
kfreebsd /root@/boot/kernel/kernel
kfreebsd /root@/boot/zfsloader
kfreebsd /root@/boot/loader
kfreebsd //@/boot/kernel/kernel
kfreebsd //@/boot/zfsloader
kfreebsd //@/boot/loader
kfreebsd /boot@/kernel/kernel
kfreebsd /boot@/zfsloader
kfreebsd /boot@/loader
kfreebsd /ROOT/default@/boot/kernel/kernel
kfreebsd /ROOT/default@/boot/zfsloader
kfreebsd /ROOT/default@/boot/loader

Every variant I've tried results in the same error:
Code:
error: unknown filesystem.

If I prefix any of the above variants with (hd0,gpt8) (i.e., my freebsd-boot partition), I get the same error. This makes me think that GRUB is able to detect my freebsd-boot partition, but doesn't know what to do with it.

If I prefix any of the above variants with (hd0,gpt9) (i.e., my ZFS /boot partition), I get the following error:
Code:
error: file not found.

If I use "chainloader +1" instead of a kfreebsd line, I get:
Code:
error: invalid EFI file path.

For all my testing, I'm using:
Code:
insmod bsd
insmod geli
insmod part_bsd
insmod part_gpt
insmod ufs1
insmod ufs2
insmod zfs
insmod zfscrypt
insmod zfsinfo

On Linux, whenever I run sudo grub-mkconfig -o /boot/grub/grub.cfg, it's unable to detect FreeBSD at all.

I'm using GRUB 2.02.beta2-4.
 
Last edited by a moderator:
GRUB2, even when compiled from source repository cannot mount a ZFS root. This broke for GRUB sometime in March or April 2014 when FreeBSD's ZFS code was upgraded. The new ZFS features are not recognized by GRUB. You can always check this problem with:
Code:
# grub-probe -d /dev/ada0p<n>
grub-probe: error: unknown filesystem.
That leaves you with two alternatives:
  • Place the contents of /boot on a UFS or EXT2 partition, mount the ZFS root from the FreeBSD kernel.
  • Compile your own GRUB from: https://github.com/dweeezil/grub/commits/zfs. This is a branch from GRUB-proper, but you will have to wait a while for this option. There was a compile error on FreeBSD which was corrected on the main tree but that correction has not made it to this branch as yet (so compilation will most likely fail).
BTW, that partition table is a mess. I had a nice detailed thread explaining how to configure partitions for multi-boot, but I can't seem to find it ATM.
 
Last edited by a moderator:
On my system GRUB2 boots ZFS on FreeBSD 10 too. GRUB2 is installed into Linux Mint 17.

apt-cache showpkg grub2
Code:
Package: grub2
Versions:2.02~beta2-9ubuntu1 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-amd64_Packages)

The menu entry code for FreeBSD is:
Code:
menuentry "FreeBSD 10.0" {
  insmod zfs
  search --set=root --label zroot --hint hd0,msdos2
  kfreebsd /@/boot/kernel/kernel
  kfreebsd_module_elf /@/boot/kernel/opensolaris.ko
  kfreebsd_module_elf /@/boot/kernel/zfs.ko
  set kFreeBSD.vfs.root.mountfrom=zfs:zroot

It's a single MBR partition with boot and root into a ZFS filesystem. The zpool is named "zroot".
 
Interesting:
Code:
# grub-probe -d /dev/sda8
grub-probe: error: unknown filesystem.
# grub-probe -d /dev/sda9
zfs
# grub-probe -d /dev/sda10
grub-probe: error: unknown filesystem.
# grub-probe -d /dev/sda11
grub-probe: error: unknown filesystem.

My freebsd-zfs /boot partition is detected as ZFS; however, my freebsd-zfs / partition is detected as an unknown filesystem. Here's how I set up the disks:

Code:
gpart add -b <start> -t freebsd-boot -s 512K -a 4K <device>
gpart add -b <start> -t freebsd-zfs -s 1G -a 4K <device>
gpart add -b <start> -t freebsd-swap -s 4G -a 4K <device>
gpart add -b <start> -t freebsd-zfs -a 4K <device>
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i <index_of_freebsd-boot_partition> <device>
kldload geom_eli
kldload zfs
geli init [...] <root_partition>
geli attach <root_partition>
zpool create -m /mnt zroot <encrypted_root_partition>
zpool create -m /mnt/boot zboot <boot_partition>

Actually, it looks like the file system type probably isn't being detected because the partition is GELI encrypted. That's my guess.

Before exiting the installer, I added
Code:
geom_eli_load="YES"
to /boot/loader.conf and
Code:
zfs_enable="YES"
to /etc/rc.conf.

I used FreeBSD-10.0-RELEASE-amd64-memstick.img.

Based on everyone's replies, I plan on reinstalling FreeBSD tonight using UFS for /boot.
 
Sure. GRUB has no GELI support. GRUB does not need access to the root partition, only /boot. Can you try this on the GRUB command line?
Code:
insmod zfs
search --set=root --label zboot --hint hd0,gpt9
kfreebsd /@/kernel/kernel
boot
This should boot the kernel and lead you to a "mountroot>" prompt.
 
Can you try this on Grub Command line ?
Code:
insmod zfs
search --set=root --label zboot --hint hd0,gpt9
kfreebsd /@/kernel/kernel
boot
This should boot the kernel and lead you to a "mountroot>" prompt.

If I try that on the GRUB command-line, my computer becomes unresponsive. Here's what the debug output is just before my machine becomes unresponsive:
Code:
mmap/efi/mmap.c:66: EFI memory region 0xff000000-0x100000000: 11
loader/i386/bsd.c:398: 22 entries in smap
lib/relocator.c:1239: min_addr = 0x17f0f90, max_addr = 0xffffffffffffffff, target = 0x17f1000
lib/relocator.c:434: trying to allocate in 0x17f1000-0xffffffffffffffff aligned 0x1 size 0x4000
lib/relocator.c:1186: allocated: 0x17f1000+0x4000
lib/relocator.c:1275: allocated 0x17f1000/0x17f1000
lib/relocator.c:1293: relocators_size=12
lib/relocator.c:1301: relocators_size=12
lib/relocator.c:1308: cur = 0xc8db4dc0, next = 0xc716d860

If I try the same thing by editing the FreeBSD menu entry instead of dropping into the command-line, then my computer either restarts or hangs (depending on its mood) when I try to boot the entry.
 
I installed ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RC2-amd64-uefi-memstick.img using freebsd-ufs for /boot.

Trying to boot FreeBSD via GRUB still makes my machine unresponsive: black screen and won't respond to keyboard input (hitting caps lock doesn't even light up the LED on the keyboard).

Here's my /boot/loader.conf:
Code:
geom_eli_load="YES"
zfs_enable="YES"
vfs.root.mountfrom="zfs:zroot"

My /etc/rc.conf:
Code:
zfs_enable="YES"
geli_devices="ada0p11"

When I boot into the installer to drop into a shell to debug, I can geli attach ada0p11 just fine. When I do zpool list, it doesn't list anything. When I do zpool import, I see zboot and zroot. I'm surprised to see zboot, since it should no longer exist (I deleted all the partitions and redid everything when I installed FreeBSD 10.1). If I do zfs mount -a, nothing appears under /mnt (-m /mnt is what I used when I created the zpool). If I do zpool import zroot, it complains that zroot may already be in use, so I do zpool import -f zroot, and it finally gets mounted under /mnt.

My expectation is that I boot FreeBSD via GRUB, which takes me to the FreeBSD loader, which asks me for my passphrase to unlock my GELI-encrypted partition, which allows the zpool to be mounted.

I think it's worth noting that even though freebsd-boot is partition 8, gpart show ada0 shows that it's at the end of the disk. Here's the order that it shows:
Code:
ada0p1
ada0p2
ada0p3
ada0p4
ada0p5
ada0p6
ada0p9
ada0p10
ada0p11
ada0p7
ada0p8

Is it necessary to have freebsd-boot if we're just hitting freebsd-ufs directly?

I tried building GRUB from source (git://git.savannah.gnu.org/grub.git) and booting using that GRUB, but booting the FreeBSD menu entry still makes my machine unresponsive.

I'm not sure if I'm hitting a bug of some sort or if I messed something up during install (I've already posted exactly how I set up my disks). Please let me know if there's anymore info that would be helpful for you to know. I can't wait to use FreeBSD. :(
 
On my System Grub2 boots ZFS on FreeBSD 10
That just means your zpool is not at the latest version. zpool upgrade -v and zpool upgrade <tank> will give you some idea.
I tried building GRUB from source (git://git.savannah.gnu.org/grub.git)
As I had stated, that repository was able to boot ZFS up until some time April. The subsequent ZFS feature updates since then have broken GRUB's ability to detect the file-system type for the last two ZFS version upgrades.

A code fix for this problem will most likely come from the ZFS-on-Linux folks and they are aware of the issue. The link I posted is to a tree with supposedly updated ZFS code for GRUB, but again as I had stated, the code patch for compile failure on FreeBSD has not been integrated to that tree as yet, so you (and I) will have to wait to test that.

GELI: I have no experience with the issue, but I think GELI support is enabled in GRUB2, because a) grub/i386-pc has geli.mod and b) https://chromium.googlesource.com/chromiumos/third_party/grub2/+/0b2d29898512e97045bec6103efd283762b8a241/grub-core/disk/geli.c So, make sure you have insmod geli in your grub.cfg.

Here's my /boot/loader.conf
loader.conf is read by the BTX loader, which means it's not read when you boot directly from GRUB, hence the kfreebsd_module_elf business in grub.cfg. The file loader.conf is read if you boot via GRUB's chainloader, which is also a fine way (and probably easier considering GELI) to do it. You'll need to add some stuff like:
Code:
kfreebsd_module_elf /@/boot/kernel/geom_eli.ko
To repeat: Direct-booting -> configure your grub.cfg correctly. Chainload-booting -> use GRUB to chainload to BTX and configure loader.conf. These are mutually exclusive.

I'm surprised to see zboot, since it should no longer exist
# zpool labelclear <device>

Can you post how many zpools you have? It looks like you have four! Am I to understand you have GRUB on ZFS as well? What are each of your partitions assigned to? Your code above show that you created a second swap partition? How are you sharing the /home folder between OS?

Please review this thread on HDD partitioning with multi-boot: https://forums.freebsd.org/threads/grub-loader-overwritten.31713
 
Code:
# zpool upgrade zroot
This system supports ZFS pool feature flags.

Pool 'zroot' already has all supported features enabled.

When I use "chainloader +1", I get:
Code:
error: invalid EFI file path.

My guess is that that's because I'm using an EFI aware GRUB, which must be interpreting the argument to chainloader as an EFI file path.

Here's what my grub.cfg currently is:
Code:
menuentry 'FreeBSD 10.0' {
    insmod geli
    insmod zfs
    insmod ufs2
    insmod bsd
    set root=(hd0,gpt9)
    kfreebsd /kernel/kernel
    kfreebsd_module_elf /kernel/geom_eli.ko
    kfreebsd_module_elf /kernel/zfs.ko
    kfreebsd_module_elf /kernel/opensolaris.ko
}

I can't put kfreebsd_module_elf first because:
Code:
error: you need to load the kernel first.

The kfreebsd line is what makes my machine unresponsive.

zpool labelclear -f /dev/ada0p9 got rid of zboot, so it no longer appears in the zpool import output. Thanks! However, I noticed that grub can't detect a file system on the partition anymore, even though it was a UFS2 partition.

I only had two zpools before today: zboot and zroot. When I installed FreeBSD 10.1 from scratch today, I only created one zpool: zroot. GRUB is on my EFI System Partition (VFAT). Windows came with 5 partitions. I'm only using one ext4 partition for Linux (no swap at all for Linux). I'm using 4 partitions for FreeBSD. Since I'm using GELI encryption, I need to keep /boot and / separate; I want swap; and FreeBSD requires that I have a freebsd-boot partition. I'm not sharing a home directory between any of the OSs. I now only have one ZFS partition (I used to have two).

Here's what I want to try next: reinstall FreeBSD 10.1 from scratch again, except this time, zpool destroy and zpool labelclear everything, so that ZFS doesn't affect future installations. If that fails to resolve the issue, then I want to reinstall FreeBSD 10.1 from scratch (doing the ZFS cleanup first again), except this time, don't use GELI at all. If I can boot FreeBSD via GRUB with root on ZFS without GELI, then I'll at least know that GELI is involved with the issue. If I still can't boot, then it's back to the drawing board.

I can also try building https://github.com/dweeezil/grub.git from source and booting from that version of GRUB.
 
I have tried to create a similar configuration (EFI+GPT+GRUB2+Linux+FreeBSD) on Virtualbox. I have installed FreeBSD with a UFS /boot and a freebsd-boot partition. Installing FreeBSD in EFI is impossible, but we can switch the virtual machine to BIOS mode during installation.

Back in EFI mode, I have tried to boot FreeBSD in two ways:

Code:
set root=(hd0,gpt4)
chainload +1
As freebeastie already said, it fails because it needs an EFI file.

Code:
insmod ufs2
set root=(hd0,gpt5)
kfreebsd /kernel/kernel
boot
The virtual system gets stuck after the boot command too.

That leads me to the central question: Is it possible to boot FreeBSD in a pure EFI mode at all?
 
Is it possible to boot FreeBSD in a pure EFI mode at all?
Now I can answer myself:
10.0: No
10.1: Yes

10.1 has now a loader.efi file, who can be used for chainloading.
Code:
insmod ufs2
set root=(hd0,gpt5)
chainloader /boot/loader.efi
boot
This loader needs all files in a /boot/ sub-directory, so you must create one and move all files to it.

Too bad the kernel also crashes this way, but I think this is more related to Virtualbox (it's EFI is in experimental state too) than to FreeBSD.
 
Just to repeat here what I had stated in that linked thread, my preferred partition structure for multi-booting with grub would be:
Code:
ada0p10  bios-boot  (64K)
ada0p1   swap
ada0px     Various OS
ada0py   zfs   home+all_data
ada0p9   ext2fs   grub   (512M)

What is this saying in comparison to your setup then?
  • Swap location is preferred near the beginning of HDD (for speed)
  • A separate home is preferred for data access through different OS
  • A separate GRUB partition is essential.
  • I use the same partition numbering scheme on all my devices, which makes it simpler to remember. You can achieve this with the -i index flag when using gpart add.
  • I would look into re-claiming some of the windows-occupied HDD space
Now about the errors in your setup:
FreeBSD requires that I have a freebsd-boot partition
No, the BTX boot loader requires that. Grub requires bios-boot partition with the boot flag toggled. You're using BTX code to boot into grub then back to BTX/FreeBSD (or whatever it is, I can't figure it out)
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot
Again, you should not be using this with grub. Let grub do its own thing and get you to the grub menu without 2nd, 3rd, 4th party modifications.

* In your Linux system (where I assume you have GRUB installed?), create the GRUB partition as I have advised, format to EXT2 (not 4) and mount it on /boot/grub (not /boot). Change partition type "freebsd-boot" to "bios-boot" and toggle the boot flag.
# grub-install --no-floppy /dev/sda
# grub-mkconfig -o /boot/grub/grub.cfg

EXT2 because FreeBSD does not have decent EXT4 r/w support, while the same goes for UFS on Linux. This should not be ZFS by any means.

* Now on to this GRUB partition, create a new folder and copy the contents of FreeBSD's /boot to (/dev/ada0p9)/bsd. Create a GRUB menu entry as below. Although you are not explicitly using chainloader, you are in fact booting via chainloading to another loader.
Code:
menuentry 'FreeBSD' {
  set root='hd0,gpt9'
  kfreebsd /bsd/loader(.efi? I don't know)
}
In the FreeBSD /, you'll have to mount the GRUB partition somewhere (/grub?) and provide a symlink to /grub/bsd named boot. This is a work-around until you get a ZFS-capable GRUB.

* Fix your /boot/loader.conf as needed, for example:
Code:
zfs_load="YES"
opensolaris_load="YES"
vfs.root.mountfrom="zfs:zroot"
geom_eli_load="YES"

In my previous post I did not mean to suggest that you should use "chainloader +1" syntax - my mistake. I mean that "kfreebsd /boot/loader" is a type of chainloaded booting of bootloaders.

I can also try building https://github.com/dweeezil/grub.git from source
Does not compile in FreeBSD. Might on Linux.

Once you get all this sorted out, we can then look into streamlining the setup with ZFS direct boot and GELI problems.
 
Ordoban, thanks for taking the time to experiment with the same configuration!

Beeblebrox, based on my understanding of your post, here's what I did:
  1. On Linux:
    • Use parted to create an EXT2 partition.
    • Use mkfs.ext2 to create an EXT2 file system on the EXT2 partition.
    • Create a grub directory under /boot.
    • Mount the EXT2 partition to /boot/grub.
    • Create a bsd directory under /boot/grub.
    • grub-install
    • grub-mkconfig -o /boot/grub/grub.cfg
  2. In the FreeBSD 10.1 installer:
    • gpart add -b <start> -t bios-boot -s 64K -a 4K <device>
    • gpart set -a bootme <bios-boot>
    • gpart add -b <start> -t freebsd-swap -s 4G -a 4K <device>
    • gpart add -b <start> -t freebsd-zfs -a 4K <device>
    • kldload geom_eli
    • kldload zfs
    • geli init -b -s 4096 -l 256 <root_partition>
    • geli attach <root_partition>
    • zpool create -m /mnt zroot <encrypted_root_partition>
    • mkdir /mnt/grub
    • mount -t ext2fs <grub_partition> /mnt/grub
    • ln -s /mnt/grub/bsd /mnt/boot
    • Continue with the FreeBSD installation.
I think we're making progress, but I don't think we're there yet.

Code:
# kfreebsd /bsd/kernel/kernel
<my system becomes unresponsive>

Code:
# kfreebsd /bsd/loader
<my system becomes unresponsive>

Code:
# kfreebsd /bsd/loader.efi
error: invalid a.out header.

Code:
# chainloader /bsd/kernel/kernel
error: cannot load image.

Code:
# chainloader /bsd/loader
error: cannot load image.

Code:
# chainloader /bsd/loader.efi
<I'm taken to the FreeBSD/amd64 EFI loader, but it says "can't load 'kernel'">
# boot
can't load 'kernel'
no bootable kernel

I can look into why the FreeBSD loader can't load 'kernel' or why using kfreebsd /bsd/loader.efi results in error: invalid a.out header. Those seem to be the most promising things to explore.
 
Last edited by a moderator:
When in the FreeBSD EFI loader, ls shows open '/' failed: no such file or directory. lsdev shows all my partitions. It seems that FreeBSD doesn't support booting from ext2, so I reinstalled FreeBSD 10.1 yet again, except this time, I used freebsd-ufs for /boot like I had before. The difference is that this time I'm using chainloader /loader.efi instead of kfreebsd /loader. This takes me to the FreeBSD EFI loader, which still shows me can't load 'kernel'; however, this time, ls shows me the contents of /boot. If I issue boot kernel/kernel, I'm eventually taken to a mountroot> prompt. I'm not sure what to do at the mountroot prompt to attach my GELI device.

My grub.cfg looks like:
Code:
menuentry 'FreeBSD 10.0' {
    insmod geli
    insmod zfs
    insmod ufs2
    insmod bsd
    set root=(hd0,gpt9)
    chainloader /loader.efi
    kfreebsd_loadenv /device.hints
    kfreebsd_module_elf /kernel/geom_eli.ko
    kfreebsd_module_elf /kernel/opensolaris.ko
    kfreebsd_module_elf /kernel/zfs.ko
    set kFreeBSD.vfs.root.mountfrom=zfs:zroot
}

I see the following when I try to boot the FreeBSD menu entry:
Code:
    Booting a command list

/EndEntire
file path: /ACPI(...,0)/PCI(2,1f)/Sata(0,ffff,0)/HD(9,...,...,...,2,2)/File()/File(loader.efi)/EndEntire
error: only FreeBSD supports environment.
error: only FreeBSD supports module.
error: only FreeBSD supports module.
error: only FreeBSD supports module.

Press any key to continue...
<After pressing any key to continue, or after a few seconds have elapsed, I'm taken to the FreeBSD EFI loader>

Does error: invalid a.out header indicate a FreeBSD bug? I would like to use kfreebsd /loader.efi; however, I can't, due to the invalid a.out header error.

At this point, I think I need to work out the kinks in my grub.cfg (see above), my /boot/loader.conf:
Code:
geom_eli_load="YES"
zfs_load="YES"
zfs_enable="YES"
opensolaris_load="YES"
vfs.root.mountfrom="zfs:zroot"

and my /etc/rc.conf:
Code:
zfs_enable="YES"
geli_devices="ada0p11"

I'm also curious why the FreeBSD EFI loader can't load my kernel by itself; however, if I do boot kernel/kernel manually, it does load.
 
In your first post, steps 1 and 2: You need to run grub-install /dev/sda AFTER having set bootme bios-boot. GRUB places its boot code on the bios-boot partition.

For the sym-linking, the logic is correct, but check whether the currect link points to the right folder, considering that zroot is no longer mounted on /mnt. Better code would have been:
# cd /mnt
# ln -s grub/bsd boot

It seems that FreeBSD doesn't support booting from EXT2
If you got R/W support for UFS in Linux, that's great! No need for EXT2 then. However, I'm not sure whether or not you have mistakenly created two separately populated /boot folders (one on grub/bsd, the other on zroot)? Make sure you have a clear map of what is where in your head:
(hd0,gpt9)/bsd/kernel is where your kernel and modules are, and that's all the files you need to boot the kernel. Your root is somewhere else, and mount_root is a different process than kernel_boot. You are not using anything on zfsroot for booting the kernel.

This takes me to the FreeBSD EFI loader, which still shows me can't load 'kernel'; however, this time, ls shows me the contents of /boot
Again, I assume you mean contents of the bsd folder? This is the BTX loader interface, and you can use the btx_command_line (see loader(8)()) to manipulate the next steps:
> load kernel/geom_eli.ko
> boot kernel/kernel

mountroot> prompt means the kernel has booted. "?" will show you available root paths.

You don't need any kfreebsd entries in grub.cfg. As I stated, when booting to BTX, the settings there have no significance.

my /boot/loader.conf
Again, where is this located? The real path (from Grub's perspective) should be (hd0,gpt9)/bsd/loader.conf

EDIT: For EFI issues, a recent thread has some insight. I know you are on 10, but the bootloader code rarely changes so information in the thread has relevance: http://freebsd.1045724.n5.nabble.com/CURRENT-EFI-boot-failure-td5949387.html
 
Last edited:
I'm not sure whether or not you have mistakenly created two separately populated /boot folders (one on grub/bsd, the other on zroot)?

Before each reinstall, I blow everything away, including doing an rm -rf /boot/grub. My FreeBSD /boot is completely on its own freebsd-ufs partition. I don't have it mounted it on or symlinked to /boot/grub/bsd like what I did when I had an EXT2 partition.

This is the BTX loader interface, and you can use the btx_command_line (see loader(8)()) to manipulate the next steps:
> load kernel/geom_eli.ko
> boot kernel/kernel

The loader says I need to load the kernel before I can load any kernel modules.

mountroot> prompt means the kernel has booted. "?" will show you available root paths.

"?" will show me all my partitions; however, I don't see ada0p11.eli and I can't seem to mount ada0p11 due to
Code:
error: unknown filesystem
I'm stuck on how to attach the GELI device at the mountroot> prompt.

You don't need any kfreebsd entries in grub.cfg. As I stated, when booting to BTX, the settings there have no significance.

That's a relief. That's one less file to worry about now. :)

The real path (from Grub's perspective) should be (hd0,gpt9)/bsd/loader.conf

Once I ditched EXT2 for FreeBSD UFS again, I no longer have /boot/grub/bsd, which was on my ESP. The contents of the FreeBSD /boot are accessible via (hd0,gpt9)/, so in GRUB, I use
Code:
chainloader (hd0,gpt9)/loader.efi
and in the FreeBSD EFI loader, I can just do an ls, which shows me the contents of /boot.

EDIT: For EFI issues, a recent thread has some insight. I know you are on 10, but the bootloader code rarely changes so information in the thread has relevance: http://freebsd.1045724.n5.nabble.com/CURRENT-EFI-boot-failure-td5949387.html

Thanks for the link. It sounds like explicitly using text mode is something worth looking into.
 
Last edited by a moderator:
"?" will show me all my partitions; however, I don't see "ada0p11.eli
That's because you have not loaded the geli module.
Code:
> load kernel/kernel
> load kernel/geom_eli.ko
> load kernel/zfs.ko
> load kernel/whatever_else.ko
> boot

Once I ditched ext2 for freebsd-ufs again, I no longer have /boot/grub/bsd, which was on my ESP. The contents of the FreeBSD /boot are accessible via (hd0,gpt9)/

I don't understand - so where is the GRUB code then? Which partition is this /boot on?

The boot process on your PC (neglecting the other OS available through GRUB):
  1. BIOS/EFI ->
  2. bios-boot (hd0,gpt10) ->
  3. GRUB (hd0,gpt9) ->
  4. FreeBSD BTX (hd0,gpt9)/bsd ->
  5. Boot kernel: FreeBSD_kernel_folder (hd0,gpt9)/bsd/kernel ->
  6. Mount root (wherever). Specify all mount settings and kernel modules in (hd0,gpt9)/bsd/kernel/loader.conf
4 should work directly, because once you get to GRUB, the work of BIOS/EFI is finished, done. From then on control has been passed to software. GRUB is EFI-capable, otherwise it would not have booted in the first place. GRUB does not need any further EFI code to boot other systems.

Once you get to the GRUB menu, all discussions then-on with regards to FreeBSD are how to mount zroot and how to get GELI working. For the time being we are simplifying by going through BTX. Get the simple boot case working on GRUB -> BTX, then you can proceed to direct booting from GRUB without the use of BTX.

You're mixing a bunch of separate concepts into a single mess of a problem, and you're skipping over things that I had previously suggested. Slow down and digest it.

* Does the code in bios-boot belong to GRUB or some other program?
 
Solution

Thanks to Beeblebrox and Ordoban for all the patience, help, and insights. It's been a frustrating ordeal, but at least I learned a thing or two during the process.

In post #14, Ordoban said "This loader needs all files in a /boot/ sub-directory, so you must create one and move all files to it." He clarified this for me in a PM. He said that "The loader.efi mounts the boot-partition as /, so it sees /kernel/kernel, /kernel/(modules...), /loader.efi, /loader.conf, etc., but loader.efi expects to see /boot/kernel/kernel, /boot/kernel/modules(...), /boot/loader.efi, /boot/loader.conf, etc."

He then recommended the following:
Code:
mount /dev/ada0pN /mnt
cd /mnt
mkdir boot_tmp           (there exists a file named boot, so we must chose a different name)
mv * boot_tmp/           (the error message "cannot move boot_tmp.." we can ignore)
mv boot_tmp boot         (now the boot file is moved, we can rename the directory)
cd ..
umount /mnt

This did the trick for me. I can now boot my GELI-encrypted root on ZFS FreeBSD system via GRUB2. I'm prompted for my passphrase to unlock my GELI device, and I'm eventually taken to a login prompt, where I can log in as my FreeBSD user.

Since there were a lot of posts, here's a summary of my final setup:

I have Windows 8, Arch Linux, and FreeBSD 10.1-RC2 living in harmony on the same disk. Everything is using GPT, not MBR.

Secure Boot, Quiet Boot, and Fast Startup are all disabled.

GRUB2 is my boot manager. I'm using the one from Linux, so when I need to update GRUB, or install it to ESP, or update the configuration files, that's where I do it. I'm using GRUB 2.02.beta2-4. I'm able to boot all three operating systems.

I skipped the gpart bootcode step entirely. I don't need it since I'm using GRUB2 to chainload the FreeBSD boot loader directly.

My partitions:
Code:
ada0p1     400M      NTFS                  (Windows)
ada0p2     300M      VFAT                  (EFI System Partition)
ada0p3     128M      Microsoft reserved    (Windows)
ada0p4     99.7G     NTFS                  (Windows)
ada0p5     350M      NTFS                  (Windows)
ada0p6     400G      ext4                  (Linux)
ada0p9     1G        freebsd-ufs: /boot    (FreeBSD)
ada0p10    4G        freebsd-swap          (FreeBSD)
ada0p11    408.7G    freebsd-zfs: /        (FreeBSD)
ada0p7     17G       NTFS                  (Windows)
ada0p8     64K       bios-boot [bootme]    (FreeBSD)

I'm not sure if I even need a bios-boot partition. I see that grub-install installs to my EFI System Partition (ESP): ada0p2. It would be nice to share a /home between operating systems, but I'm using encryption with Linux and encryption with FreeBSD, so I'm not sure if that's feasible or even possible to share a home between the two. I don't want to have an unencrypted /home. I have the same problem with swap (except that I'm not even using swap with Linux). I definitely would like to clean up the Windows mess, but I'm not too familiar with Windows, so I'm worried if I move partitions around or remove any partitions, I might hose the system.

If FreeBSD is able to write to VFAT, then I might be able to just install FreeBSD /boot to the ESP (ada0p2) and do away with the freebsd-ufs partition.

My /boot/grub/grub.cfg (made from /etc/grub.d/40_custom, which is where my FreeBSD menu entry is defined, and others):
Code:
menuentry 'FreeBSD 10.1-RC2' {
    set root=(hd0,gpt9)
    chainloader /boot/loader.efi
}

Note that my gpt9 is my freebsd-ufs /boot partition. I found that I didn't need to insmod any modules in GRUB. The two-liner menuentry was enough for me. You might even be able to get away with a one-liner: chainloader (hd0,gpt9)/boot/loader.efi.

My /etc/rc.conf:
Code:
zfs_enable="YES"
geli_devices="ada0p11"
geli_swap_flags="..."

My /boot/loader.conf:
Code:
geom_eli_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

The only problem I see is a bunch of USB messages littering the screen. This mainly causes a problem with prompts (it's easy to miss a prompt and hard to see input when the USB messages keep appearing); however, I have the same problem with Linux. I know that the USB messages are harmless. I'm sure there's a way to suppress the messages.
Code:
ugen0.4: <Generic> at usbus0
usbd_setup_device_desc: getting device descriptor at addr 4 failed, USB_ERR_TIMEOUT
ugen0.5: <Unknown> at usbus0
uhub_reattach_port: could not allocate new device
ugen0.2: <PixArt> at usbus0

I hope this helps someone, and I hope this process becomes smoother in the future (I'm sure it will). :)

Thanks again for all the help Beeblebrox and Ordoban! I'm relieved and happy that I finally get to enter into the world of FreeBSD. Time to have some fun.

UPDATE:

It is possible to boot GELI-encrypted root on ZFS with /boot on ESP! I now have just two FreeBSD-only partitions: one for root and one for swap. I don't need a bios-boot partition; I don't need a freebsd-boot partition; and I don't need a separate partition for /boot other than my pre-existing ESP, which is shared among all the operating systems on my UEFI disk. I also managed to get rid of the 17 GiB NTFS partition.

Essentially, you need to make sure your ESP is big enough and you need to make sure there is a boot directory at the root of the ESP and keep all the BSD boot stuff there. In my case, 600 MiB would have been just enough, but I decided to go for 1 GiB, just to have some breathing room.

It would not be very wise of you if you decide to proceed with the following steps without having backed up your data (especially your ESP) and without having a LiveCD handy and without having recovery disks prepared (such as for Windows). Make sure you have safety nets in place all along the way. You may very well need them, and you'll love yourself for making your life easier.

Now, the idea is simple, but in my case, I had to create a new ESP, which complicated things. I wanted to share what steps I took, in case they prove useful to anyone else. Obviously, not all the steps will apply to everyone, but I hope you can get a sense for what you need to do, and then generalise to your situation. Here we go:
  1. Back up the data on your disk to an external disk.
  2. Back up your EFI System Partition (ESP). This is extremely important.
  3. Create a Windows 8.1 recovery USB disk (Windows 8.1 doesn't support making a recovery CD).
    1. In Windows, search for "Create a recovery drive" and then select the search result.
    2. Tick the "Copy the recovery partition from the PC to the recovery drive" option.
    3. Next.
    4. Next.
    5. Create.
    6. Delete the recovery partition.
    7. Delete.
      1. Obviously, only do this if you actually want to delete the recovery partition.
    8. Finish.
  4. Using whatever partition editor on whatever operating system, create a new FAT32 partition in the free space on your disk. This will be your new ESP. I made mine 1 GiB. You might get away with 500 MiB if you only have two operating systems on the disk, but the more operating systems you have, the more space you should allocate.
  5. Remove the boot flag from your old ESP.
  6. Assign the boot flag to your new ESP.
  7. Copy the contents of your old ESP to your new ESP.
  8. Update the /etc/fstab on all your machines to mount your new ESP instead of your old one.
At this point, I decided to delete my old ESP. The Windows boot manager expected things to be on the old ESP, so using the Windows boot manager directly no longer worked, but I could still boot Windows via GRUB. I decided to fix the Windows boot manager, but Windows isn't aware (or doesn't care) that it shares a disk with other operating systems. This meant that GRUB was removed from NVRAM, which meant that I could no longer boot into GRUB. The cherry on top is that a lot of LiveCDs boot into BIOS mode, which means they don't have efivars, which means you can't fix GRUB.
  1. Delete your old ESP.
  2. Boot your Windows 8 recovery media.
    1. Choose your keyboard layout.
    2. Click Troubleshoot.
    3. Click Advanced Options.
    4. Click Command Prompt.
    5. Look up the path to your Windows boot manager.
      • bcdedit /v
      1. Under Windows Boot Manager, look at the value of path and write it down. In my case, the path was \EFI\Microsoft\Boot\bootmgfw.efi.
    6. diskpart
    7. list disk
    8. sel disk <N>
    9. list vol
    10. sel vol <new_ESP>
    11. assign letter=<drive_letter_for_ESP>
    12. exit
    13. cd /d <drive_letter_for_ESP>:\EFI\Microsoft\Boot
    14. ren BCD BCD.old
    15. bootrec /fixboot
    16. bootrec /fixmbr
    17. bootrec /rebuildbcd
    18. bcdboot <drive_letter_for_Windows>:\Windows /l <locale> /s <drive_letter_for_ESP>: /f ALL
    19. exit
    20. Click Continue.
  3. Boot your LiveCD.
    1. Mount your ESP.
    2. cd into your ESP.
    3. cd EFI
    4. cp -r Microsoft OldMicrosoft
    5. cp GRUB/grubx64.efi Microsoft/Boot/bootmgfw.efi
    6. Unmount your ESP, remove the LiveCD from your machine, and then reboot.
    7. Once you've booted into the system where GRUB is installed, fix your GRUB by doing grub-install followed by grub-mkconfig like you normally would.
    8. cp OldMicrosoft/Boot/bootmgfw.efi Microsoft/Boot/bootmgfw.efi
Some people say that you can use bcdedit /set {bootmgr} path \EFI\GRUB\grubx64.efi to boot into GRUB, but I found that that didn't work for me. Maybe it would have worked if I had redone the bootrec commands (see above), but I found it was just easy enough to rename the GRUB boot loader to \EFI\Microsoft\Boot\bootmgfw.efi temporarily.

Now, when you install FreeBSD:
  1. Mount your root partition to /mnt.
  2. mkdir /mnt/esp
  3. Mount your ESP to /mnt/esp.
    • mount -t msdosfs /dev/<ESP> /mnt/esp
  4. cd /mnt
  5. mv -i esp/Boot esp/OldBoot
  6. mkdir esp/boot
  7. ln -s esp/boot boot
  8. exit
Proceed with installation as normal. Before rebooting, don't forget to modify your /etc/rc.conf, your /boot/loader.conf, or your /etc/fstab, if necessary. If your GRUB has a FreeBSD boot entry that has chainloader /boot/loader.efi with root set to your ESP, then you should now be able to boot into FreeBSD via GRUB.
 
Last edited:
I'm posting the below to share my grub2 experience and answer some questions posted in this thread, but also to find some answers you could possibly give me.
GRUB2, even when compiled from source repository cannot mount a ZFS root...
You are using the Linux version of GRUB2, that's why. If you use sysutils/grub2 instead, you will have access to ZFS volumes and everything else. One more area where FreeBSD is superior to Linux, it seems? Then you'll be able to boot your system with it either using the kfreebsd /boot/zfsloader string, or using the more detailed configuration loading everything. Don't use UUIDs in grub.cfg, for it won't work. Use rather set root=(hdN,gptN) format.

All you need is create a bios-boot type partition with gpart, then grub-install --boot-directory /boot/grub /dev/ada0 (or whatever).

See here for more details.

For EFI boot you'll need sysutils/grub2-efi port, that's all the difference. Then you create a small partition of efi type with gpart then create a FAT16 (or 32 if it's big enough) file system there and mount it:
Code:
mount -t msdosfs /dev/ada0pN /mnt/disk
Then you can install grub2-efi there:
Code:
grub-install --modules="part_gpt zfs chain help fat gettext normal crypto configfile" --target=x86_64-efi  --efi-directory=/mnt/disk
There's currently only one problem I have with this setup: I don't know how to make the kernel use vt_efifb in this case, and by itself it uses vga, which ends up in booting with some black-white mess on the screen so you can't see the messages and have to log in by just blindly typing things and hitting Enter.
This makes it a little bit short of being "usable", so putting the whole /boot on the ESP is the only solution, right? Or is there a way to make the kernel aware that it must use EFI console? That would be great.

EDIT: I just want to emphasize the advantage of this setup: with it you don't have to put /boot on your ESP. Nor, of course, do you need a bios-boot or freebsd-boot type partitions on your GPT disk. This seems more "universal", I think... Except for this problem of kernel not being able to recognize EFI boot, unless it's called by loader.efi, and so you don't have console -- which is a really big problem, seeing that FreeBSD system itself needs console.
 
Some people say that you can use bcdedit /set {bootmgr} path \EFI\GRUB\grubx64.efi to boot into GRUB, but I found that that didn't work for me. Maybe it would have worked if I had redone the bootrec commands (see above), but I found it was just easy enough to rename the GRUB boot loader to \EFI\Microsoft\Boot\bootmgfw.efi temporarily.
................
I've noticed the same thing, and the reason for it lies in something that general EFI documentation doesn't mention. For example, /EFI/GRUB/grubx64.efi won't be recognized by my motherboard's UEFI for the required EFI software, but when renamed to /EFI/BOOT/BOOTX64.EFI it will be recognized as such and will boot without questions. I've read something to the effect, that this depends on the motherboard's UEFI, which differs from one board to another.
 
Don't use UUIDs in grub.cfg, for it won't work. Use rather set root=(hdN,gptN) format.
Well! This seems to have been fixed in sysutils/grub2, can't be sure about the grub2 source ;). So, using the string like the following:
Code:
 search --no-floppy --fs-uuid --set=root  46464c03259a8cb0
instead of set root=(hd0,gpt2) works fine now, which is what it should be, because disk numbering is a tricky thing and differs in UEFI mode from 'legacy' mode in my motherboard setup at least.

So now grub-mkconfig is almost usable, except that it insists on looking for zpool.cache in /boot/kernel/zfs instead of /boot/zfs. But that should be fixed in grub scritps, where they forget that freebsd kernel is /boot/kernel/kernel, and not /boot/kernel.

UPDATE: here is a patch to /usr/local/etc/grub.d/10_kfreebsd to fix this.
 
Last edited:
The kfreebsd line is what makes my machine unresponsive
I used to have this problem with Linux versions of GRUB2, but not any more.
I can also try building https://github.com/dweeezil/grub.git from source and booting from that version of GRUB.
Well I've tried it yesterday with git://git.savannah.gnu.org/grub.git and all works just fine, no problem loading kernel.
However, I built it on my linux system, with the last git versions of SPL and ZFS installed, against which I built the last git version of GRUB2 from the above GNU site. It works perfectly well, only I haven't tried it with the new feature flags of ZFS, which come with 10.1-RELEASE.

I'm mentioning this, because after having upgraded the pool I lost the sysutils/grub2 functionality as it cannot read some of the zpool-features(7) (namely, com.delphix:embedded_data). This makes the pool pretty much inaccessible to GRUB2 in this case, so I'm not hasting to upgrade my existing pool, where 10.1-RELEASE is installed by upgrading process, while the pool itself was created using some "obsolete" utils a year ago.

But I'll check that in the office, where I've already upgraded all the pools. Theoretically, since this build of GRUB is linked against the latest ZFS libs, I don't see why it shouldn't read those feature flags... but who knows. In any case, we always have the BTX bootloader which will boot FreeBSD even from the most advanced pool. Then there is this option to recreate the pool using some of the recent FreeBSD releases, with earlier versions of ZFS there.
 
Back
Top