bhyve nested bhyve virtual machines

ikevin8me

Active Member

Reaction score: 12
Messages: 179

I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:
vmx_modinit: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, Oxffff .....) error 6
vm_create: Device not configured

So, is there a way to make it work?
 

Argentum

Well-Known Member

Reaction score: 195
Messages: 424

I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:


So, is there a way to make it work?
I think BHYVE does not support nested virtualization this way. The virtualization hardware in the CPU is already taken by the main hypervisor.
 
OP
I

ikevin8me

Active Member

Reaction score: 12
Messages: 179

I tried to use qemu, but it does not seem to be so easy. Anydefinitive guide to install OpenBSD as guest?
 

Zirias

Daemon

Reaction score: 1,339
Messages: 2,362

Why do you want to nest?

You should keep in mind that there's no hardware supporting this. It can be done in software of course, but the performance will suffer. For more background, read for example: https://en.wikipedia.org/wiki/Second_Level_Address_Translation

A design decision for bhyve is to only work with this hardware support, so you can never nest bhyve inside a bhyve guest. Other hypervisors with the same design decision include Linux' kvm and OpenBSD's vmm.
 

sko

Aspiring Daemon

Reaction score: 348
Messages: 623

I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:


So, is there a way to make it work?

No, this is not supported as bhyve would have to emulate VT-x/SVM [1], which would definitely add a lot of bloat and has terrible performance anyways - so nobody would want to do that.

A more sane approach to contain bhyve instances would be running bhyve instances inside of jails. This approach is already used in production e.g. by ClonOS and Project-FiFo (IIRC Free/TrueNAS abandoned that approach and is/was using KVM directly on the host instead).
Using vnet-jails this is basically the same approach as smartOS is doing with bhyve/KVM inside of zones.
However, runnig FreeBSD as a bhyve guest on top of FreeBSD usually isn't what you want to do as jails are much faster and more flexible. For other guests the jail + bhyve approach adds another layer of security and it can simplify management and migration, which is still a bit fiddly with bhyve on its own. When carrying all configuration inside the jail, moving a VM to another host is basically the same as moving a jail - which nowadays is mostly automated by some tools (i.e. iocage/iocell).


[1] https://www.mail-archive.com/freebsd-virtualization@freebsd.org/msg09004.html
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

jails are much faster
Amen to that. Especially with NVMe. By the time it is passed through performance is impacted greatly.
I moved some software building from bhyve to jails just for the disk speed.
 
OP
I

ikevin8me

Active Member

Reaction score: 12
Messages: 179

I do see some security advantages to using vnet/jail + bhyve. However...

In order to run bhyve within a jail, the "vmm" needs to be passed into the jail ("add path vmm unhide", etc.). This exposes all hardware devices to the jail and bhyve. (I tried dmidecode and pciconf - and everything shows up) So, I think there is no security enhancement with regards to accessing hardware. What is your opinion?
 

Zirias

Daemon

Reaction score: 1,339
Messages: 2,362

Well, exposing one device to a jail should never automatically expose other devices as well, so something is fishy there…

But still, I don't see what good starting bhyve from inside a jail should do. The major part of the hypervisor is implemented in the kernel anyways, therefore isn't restricted by a jail.

I do agree that for running a FreeBSD "VM" on FreeBSD, most of the time a jail is the best solution. If you need completely separated networking, go for VNET. There are usecases (like e.g. running a newer version of FreeBSD, or needing exclusive access to some PCI device inside the VM) where full virtualization is necessary, but if it isn't, go with a (VNET) jail.

Repeating my question from above: Why do you want to nest?

If the answer is something with "security", you're hunting an illusion…
 
OP
I

ikevin8me

Active Member

Reaction score: 12
Messages: 179

Repeating my question from above: Why do you want to nest?
I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.
 

Zirias

Daemon

Reaction score: 1,339
Messages: 2,362

I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.
Just to make this absolutely clear: A hypervisor that allows nesting will still suffer from bad performance, because hardware only supports a single layer, see also link above about SLAT/NPT. So it's always the best idea not to nest virtual machines. There might be specific usecases for nesting (e.g. you could have a virtual machine for development and still want to test your software in a nested VM there), but you should be aware of the implications ;)
 

sko

Aspiring Daemon

Reaction score: 348
Messages: 623

I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.

yes, it can go exactly 1 layer - because more doesn't make sense.
Bhyve was designed as a fast and ultra-lightweight hypervisor by leveraging modern cpu-mechanisms for virtualization with as less software emulation as possible. If you want/need every old cruft up to FDD and ISA controllers emulated, go one of the fully-(b)loa[t|d]ed hypervisors like QEMU, VMware or Virtualbox. They drag software-emulation for every obsolete hardware since the ~mid-80s around with them, which massively increases the attack surface and overhead and allow impractical and useless features like nesting over multiple layers.
 

PMc

Daemon

Reaction score: 616
Messages: 1,263

However, runnig FreeBSD as a bhyve guest on top of FreeBSD usually isn't what you want to do as jails are much faster and more flexible.

There is one reason I can think of (and currently do think of) changing a jail to bhyve: dummynet is not working in jails, there is only one instance. (In 12.2 at least, but I have no notice that this would have improved in 13.)
Only I am not yet sure if (and how) I would route the network stuff into and out of a bhyve - but that should be workable.
 
Top