Solved FreeBSD inside bhyve

Ofloo

Ofloo

I'm trying to install freebsd as a guest os inside bhyve, without diskencryption this works just fine, however once i want to run bhyve with encrypted disks the system hangs on bhyveload

I run setup from bootonly iso then when I come to the file partition I choose option shell and setup the system as following:
Code: 
# gpart destroy -F vtbd0
vtbd0 destroyed
# gpart create -s gpt /dev/vtbd0
vtbd0 created
# gpart add -t freebsd-boot -s 512k /dev/vtbd0
vtbd0p1 added
# gpart add -t freebsd-ufs -l boot0 -s 1g -a 1m /dev/vtbd0
vtbd0p2 added
# gpart add -t freebsd-swap -l swap0 -s 2G /dev/vtbd0
vtbd0p3 added
# gpart add -t freebsd-ufs -l disk0 -a 1m /dev/vtbd0
vtbd0p4 added
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 vtbd0
partcode written to vtbd0p1
bootcode written to vtbd0
# geli init -e AES-XTS -l 256 -b -s 4096 vtbd0p4
Enter new passphrase:
Reenter new passphrase:

Metadata backup can be found in /var/backups/vtbd0p4.eli and
can be restored with the following command:

        # geli restore /var/backups/vtbd0p4.eli vtbd0p4

# geli attach vtbd0p4
Enter passphrase:
GEOM_ELI: Device vtbd0p4.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
# newfs -U /dev/vtbd0p4.eli
/dev/vtbd0p4.eli: 17406.0MB (35647480 sectors) block size 32768, fragment size 4096
        using 28 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
        with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112,
 11540352, 12822592, 14104832, 15387072, 16669312, 17951552, 19233792,
 20516032, 21798272, 23080512, 24362752, 25644992, 26927232, 28209472,
 29491712, 30773952, 32056192, 33338432, 34620672
# newfs -U /dev/vtbd0p2
/dev/vtbd0p2: 1024.0MB (2097152 sectors) block size 32768, fragment size 4096
        using 4 cylinder groups of 256.03MB, 8193 blks, 32896 inodes.
        with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 524544, 1048896, 1573248
# mount /dev/vtbd0p4.eli /mnt
# mkdir /mnt/boot
# mount /dev/vtbd0p2 /mnt/boot
# ee /tmp/bsdinstall_etc/fstab
/dev/vtbd0p2            /boot   ufs     rw      1       1
/dev/vtbd0p3            none    swap    sw,keylen=256      0       0
/dev/vtbd0p4.eli        /       ufs     rw      2       2
# ee /tmp/bsdinstall_boot/loader.conf
geom_eli_load="YES"
vfs.root.mountfrom="ufs:vtbd0p4.eli"
aesni_load="yes"

What am I doing wrong?
 
Ofloo said:
however once i want to run bhyve with encrypted disks the system hangs on bhyveload
Click to expand...
My guess is that it doesn't hang, it's just waiting for someone to enter the password to unlock the disk. I'm not sure how bhyveload(8) deals with this. You may need to attach a console and enter the password there.
 
This is where it hangs.

Code: 
Erase is backspace.
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(Mon Jul 31 19:25:12 CEST 2017 x@xxx)
-
can't load 'kernel'

Type '?' for a list of commands, 'help' for more detailed help.
OK

When I load the kernel manually, "load kernel/kernel"

This shows:
Code: 
Type '?' for a list of commands, 'help' for more detailed help.
OK load kernel/kernel
kernel/kernel text=0x14972f8 data=0x1384c0+0x4c15e8 syms=[0x8+0x15e8b0+0x8+0x178422]
OK show
LINES=24
boot_serial=1
console=userboot
currdev=disk0p2:
interpret=OK
kernelname=kernel/kernel
loaddev=disk0p2:
loader.conf=
prompt=${interpret}
smbios.bios.vendor=BHYVE
twiddle_divisor=1
OK boot
Unhandled ps2 mouse command 0xe1

And thats where it stays ..
 
I get the feeling I'm doing something wrong, .. though since when it boots normally it loads it says it's loading /boot/kernel/kernel while when I manually configure it, .. boot is in "/" maybe that's normal ..
 
What host/guest versions are you using ?

From the boot log, it looks like serial console output isn't going to stdout - you will probably need to attach to where the serial console is being directed and enter the GELI password there .
 
It is stdio and I tried using a serial console without luck, however it think the reason it doesn't boot is exactly what I said, I need to mount boot under a directory /unenc/boot then link it to the root directory because if it mounts the unencrypted boot partition the file /boot/kernel/kernel doesn't exist. Testing that theory right now :p

* It wasn't the reason :/

* on the other hand you might be right it does resemble that but i set -l com1,stdio i've tried setting it to /dev/nmdm0A but I just got the same result as I had set it to stdio

it was what I thought it was only i mounted to /unenc/boot rather then mounting to /unenc and making a directory boot then linking that directory to /boot and now it works.

Code: 
# mkdir /mnt/unenc
# mount /dev/vtbd0p2 /mnt/unenc
# mkdir /mnt/unenc/boot
# cd /mnt
# ln -s unenc/boot

So when it mounts /dev/vtbd0p2 it still has a directory /boot
 
You must log in or register to reply here.
Back
Top