Solved can't load 'kernel' - FreeBSD 11.0 with encrypted ZFS (RAID-10) and boot environments

P

pdrecker

Hi,

I like to install FreeBSD 11.0 with encrypted ZFS root and boot environments using four disks (stripped and mirrored). After the first reboot I got the following error:

Code: 
BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drives C: is disk0
BIOS drives D: is disk1
BIOS drives E: is disk2
BIOS drives F: is disk3
Calculating GELI Decryption Key disk0p3 @ 955879 iterations...
Calculating GELI Decryption Key disk1p3 @ 925444 iterations...
Calculating GELI Decryption Key disk2p1 @ 934232 iterations...
Calculating GELI Decryption Key disk3p1 @ 920264 iterations...
BIOS 638kB/3143616kB available memory

FreeBSD/x86 ZFS enable bootstrap loader, Revision 1.1
(root@releng2.nyi.freebsd.org. Thu Sep 29 01.38.49 UTC 2016)
ZFS: i/o error - all block copies unavailable

/boot/kernel/kernel test=0x14ed860 ZFS: i/o error - all block copies unavailable

elf64_loadimage: read failed
can't load 'kernel'

I tried the same installation using a virtual machine with four virtual drives (stripped and mirrored) and I got the same error. If I only use 2 drives (stripped or mirrored) everything works fine.

TEST 1 = 2 mirrored drives (works fine)
TEST 2 = 2 stripped drives (works fine)
TEST 3 = 4 stripped and mirrored drives (can't load kernel)

For this tests the partition labels have the following rules:

1) Each disk start with 'data1-'.
2) If a partition is mirrored a number (1, 2, ...) follows.
3) If a partition is stripped a character (a, b, ...) follows.
4) The last part of the label shows the usage (boot, swap, zroot, ...).

As I like to use Copy&Paste during installation I boot FreeBSD 11.0 into the 'Live CD' and enable sshd.

kbdmap
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
/etc/rc.d/sshd onestart
passwd root
ifconfig em0 up 192.168.188.11/24

To install FreeBSD 11.0 I execute the following commands:

I) TEST 1, 2 and 3

set rprompt = "%?"

II) TEST 1

gpart create -s gpt da0
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-1-boot da0
gpart add -a 4k -t freebsd-swap -s 1G -l data1-1-swap da0
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-1-zroot da0
gpart create -s gpt da1
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-2-boot da1
gpart add -a 4k -t freebsd-swap -s 1G -l data1-2-swap da1
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-2-zroot da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

II) TEST 2

gpart create -s gpt da0
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-boot da0
gpart add -a 4k -t freebsd-swap -s 1G -l data1-1-swap da0
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-a-zroot da0
gpart create -s gpt da1
gpart add -a 4k -t freebsd-swap -s 1G -l data1-2-swap da1
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-b-zroot da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

II) TEST 3

gpart create -s gpt da0
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-1-boot da0
gpart add -a 4k -t freebsd-swap -s 1G -l data1-1-swap da0
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-1a-zroot da0
gpart create -s gpt da1
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-2-boot da1
gpart add -a 4k -t freebsd-swap -s 1G -l data1-2-swap da1
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-2a-zroot da1
gpart create -s gpt da2
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-1b-zroot da2
gpart create -s gpt da3
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-2b-zroot da3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

III) TEST 1, 2 and 3

mkdir /tmp/mnt
mount_unionfs /tmp/mnt /mnt
kldload geom_eli
kldload zfs
sysctl vfs.zfs.min_auto_ashift=12
echo 'geheim' > /tmp/diskpw

IV) TEST 1

geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-1-zroot
geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-2-zroot
geli attach -j /tmp/diskpw gpt/data1-1-zroot
geli attach -j /tmp/diskpw gpt/data1-2-zroot
zpool create -o altroot=/mnt -o cachefile=none -f zroot mirror gpt/data1-1-zroot.eli gpt/data1-2-zroot.eli

IV) TEST 2

geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-a-zroot
geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-b-zroot
geli attach -j /tmp/diskpw gpt/data1-a-zroot
geli attach -j /tmp/diskpw gpt/data1-b-zroot
zpool create -o altroot=/mnt -o cachefile=none -f zroot gpt/data1-a-zroot.eli gpt/data1-b-zroot.eli

IV) TEST 3

geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-1a-zroot
geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-2a-zroot
geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-1b-zroot
geli init -g -b -e AES-XTS -l 256 -s 4096 -J /tmp/diskpw gpt/data1-2b-zroot
geli attach -j /tmp/diskpw gpt/data1-1a-zroot
geli attach -j /tmp/diskpw gpt/data1-2a-zroot
geli attach -j /tmp/diskpw gpt/data1-1b-zroot
geli attach -j /tmp/diskpw gpt/data1-2b-zroot
zpool create -o altroot=/mnt -o cachefile=none -f zroot mirror gpt/data1-1a-zroot.eli gpt/data1-2a-zroot.eli mirror gpt/data1-1b-zroot.eli gpt/data1-2b-zroot.eli

V) TEST 1, 2 and 3

zfs set atime=off zroot
zfs set checksum=fletcher4 zroot
zfs set compression=lz4 zroot
rm /tmp/diskpw
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create zroot/usr/home
zfs create -o setuid=off zroot/usr/ports
zfs create zroot/usr/src
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs create -o setuid=off zroot/var/tmp
chmod 1777 /mnt/tmp
cd /mnt
ln -s usr/home home
kldload geom_mirror
gmirror label swap gpt/data1-1-swap gpt/data1-2-swap
cd /mnt
unxz -c /usr/freebsd-dist/base.txz | tar xpf -
unxz -c /usr/freebsd-dist/kernel.txz | tar xpf -
chroot /mnt
cd /
echo 'hostname="test.foo.local"' > /etc/rc.conf
echo 'keymap="de.kbd"' >> /etc/rc.conf
echo 'zfs_enable="YES"' >> /etc/rc.conf
echo 'ifconfig_em0="up 192.168.188.11/24"' >> /etc/rc.conf
echo 'defaultrouter="192.168.188.2"' >> /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'powerd_enable="YES"' >> /etc/rc.conf
echo 'ntpd_enable="YES"' >> /etc/rc.conf
touch /etc/fstab
echo 'vfs.zfs.min_auto_ashift=12' >> /etc/sysctl.conf
echo 'vfs.zfs.prefetch_disable="1"' > /boot/loader.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'aesni_load="YES"' >> /boot/loader.conf
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'geom_mirror_load="YES"' >> /boot/loader.conf
echo 'kern.geom.eli.taste_providers="1"' >> /boot/loader.conf
echo 'kern.geom.eli.reuse_passphrase="1"' >> /boot/loader.conf
echo 'kern.geom.eli.tries="9"' >> /boot/loader.conf
echo 'kern.geom.label.gptid.enable="0"' >> /boot/loader.conf
echo 'kern.geom.label.disk_ident.enable="0"' >> /boot/loader.conf
echo 'nameserver 192.168.188.2' > /etc/resolv.conf
echo '# Device Mountpoint FStype Options Dump Pass#' > /etc/fstab
echo '/dev/mirror/swap.eli none swap sw 0 0' >> /etc/fstab
passwd root
tzsetup
echo 'nameserver 192.168.188.2' > /etc/resolv.conf
echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
cd /etc/mail
make aliases
exit
cd /
zfs unmount -a
zfs set canmount=noauto zroot/ROOT/default
zpool set bootfs=zroot/ROOT/default zroot
reboot

Has anybody an idea what I did wrong?

Regards
Pascal
 
Last edited:
Hi,

a few hours later I found a workaround.

The boot error only occurs if the ZFS partition is the first entry in the GPT. For a workaround you only have to create a tiny "dummy" partition of any type before you create the ZFS partition.

I decided to extend my mirrored swap from 2 to 4 disks.

III) TEST3

gpart create -s gpt da0
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-1-boot da0
gpart add -a 4k -t freebsd-swap -s 1G -l data1-1-swap da0
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-1a-zroot da0
gpart create -s gpt da1
gpart add -a 4k -t freebsd-boot -s 1024 -l data1-2-boot da1
gpart add -a 4k -t freebsd-swap -s 1G -l data1-2-swap da1
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-2a-zroot da1
gpart create -s gpt da2
gpart add -a 4k -t freebsd-swap -s 1G -l data1-3-swap da2
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-1b-zroot da2
gpart create -s gpt da3
gpart add -a 4k -t freebsd-swap -s 1G -l data1-4-swap da3
gpart add -a 4k -t freebsd-zfs -s 16G -l data1-2b-zroot da3
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1


If you like to use this configuration don't forget to change the command line options of gmirror:

gmirror label swap gpt/data1-1-swap gpt/data1-2-swap gpt/data1-3-swap gpt/data1-4-swap

Regards
Pascal
 
You must log in or register to reply here.
Back
Top