Solved Installing hypervisors inside jails and modding kernel

[gladiola]

Recently I began experimenting with jails to contain VMs. I got stumped when I got to the installation and initialization of VirtualBox inside a jail. My vboxheadless startup would halt because I needed to modify the kernel.

For example, on a strait host install of virtualbox, the handbook advises:

vboxdrv_load="YES"

https://www.freebsd.org/doc/handbook/virtualization-host-virtualbox.html

When we're installing hypervisors into jails, and need to modify the kernel to go with it, what is the preferred way to carry out the mod? Do I need to equip the jail host with every kernel mod needed by a jail? Do I need to follow with similar mods to the boot.conf and rc.conf of the jails that follow? Is it preferred to keep the kernel mods on the host and use the jails for the needed rc.conf?

I would generally prefer to have the jail host only run the jails. Any advice you might give could be helpful. Thanks.

 

[shkhln]

What mod? This is completely silly, jails are for userspace processes.

 

[SirDice]

Do I need to equip the jail host with every kernel mod needed by a jail?

Jails don't have a kernel. That's not how jails work. Why on earth would you want to run a virtualization layer in a jail in the first place?

 

[gladiola]

Please read the question carefully. I am aware that the jail does not have a kernel.

The jails, having a base.txz installed, carry a boot.conf. When we compile a port for virtualbox in a jail, we see the production of the expected kernel modules held in the work files of the port. vboxdrv.ko is one of those files. I saw maybe half a dozen such *.ko files in that work folder; all were related to the operation of virtualbox. Where is the best place to install those files? In the boot.conf of the jail, or the boot.conf of the host?

I want to run the virtualbox in a jail so that I can stabilize and isolate the VM. You may notice, for example, that bhyve can run inside a jail:
https://reviews.freebsd.org/D16057 My questions are about how a hypervisor can be installed properly in a jail.

As above, would the computer hosting the jail need every kernel module needed by every jail it runs loaded at the host's boot? It seems to me like it would. If so, then what becomes of the content of boot.conf in the jail? Is it just totally ignored?

So:
When we're installing hypervisors into jails, and need to modify the kernel to go with it, what is the preferred way to carry out the mod? Do I need to equip the jail host with every kernel mod needed by a jail? Do I need to follow with similar mods to the boot.conf and rc.conf of the jails that follow? Is it preferred to keep the kernel mods on the host and use the jails for the needed rc.conf?

 

[shkhln]

What do you want to achieve? Hypervisor (any kernel module, really) always runs outside of jailed environment. Same goes for VMs. Control utilities can be jailed, but I don't see any point in that.

 

[gladiola]

Please observe the example code in the link below that shows a hypervisor running inside a jail.
https://reviews.freebsd.org/rS337023

If we can return the questions outlined in the OP, I think that would be helpful. Thanks.

 

[shkhln]

You do understand the difference between hypervisor, which by definition must have the fullest possible access to the hardware, and its control utilities? Or that jails (containers) are implemented by placing appropriate access checks in syscalls?

If we can return the questions outlined in the OP

Definitely not.

 

[gladiola]

Please stop posting replies that don't answer the question. As you can see from the examples provided, hypervisors can run inside jails. This thread is about how to configure that with virtualbox.

 

[shkhln]

Your initial question doesn't make any sense, since "mod" is an abbreviation for "modification" and not for "module". These words are not synonyms and they are not interchangeable.

I don't think you can load a kernel module from a jail (without completely compromising security guaranties, that is). No idea whether you would be able to access /dev/vboxdrv from a jail either. I doubt anybody ever tried that.

 

[unitrunker]

If VirtualBox or any other jailed program requires a kernel module - that module must be loaded outside the jail (before the jail starts). Allowing a jail to muck with the kernel is contrary to the purpose of jails.

Each jail can run its own rc(8) services but loading a kernel module is not a use case for a jail. So ... to address your question:

Where is the best place to install those files? In the boot.conf of the jail, or the boot.conf of the host?

The host.

 

[_martin]

I'm guessing by the mod you do mean modification and you mean modification of the virtualbox modules the same way it was done to bhyve's vmm.ko. That question needs to be directed to virtualbox developers though.

Aside from what was said above (I agree with that) I have very bad experience with virtualbox being run by more users. I strictly use only one user that manages all VMs (user vbox) and other users in the group vboxusers are able to sudo to vbox. Why ? As a test yesterday:

FreeBSD 12.1, virtualbox 5.2.34: userA registers its own VM and runs it along with all other VMs run by vbox user. What happened ? Host-only networking got messed up (host only adapter lost its IP), VMs became unresponsive (not only due to lack of host-only networking). I powered off the VMs and attempted to restart the network (vboxnet). Not to a surprise - kernel panic. (page fault, vboxnet kernel module).
This short and boring story is very common to virtualbox on FreeBSD in my experience (not isolated to a single host).

So there is a question why would you want to have VM in jail ? If there is a reason for that maybe qemu is better option for you then.

 

[gladiola]

If VirtualBox or any other jailed program requires a kernel module - that module must be loaded outside the jail (before the jail starts). Allowing a jail to muck with the kernel is contrary to the purpose of jails.

Each jail can run its own rc(8) services but loading a kernel module is not a use case for a jail. So ... to address your question:


The host.

Found this out with some later experiments, thanks.

 

[gladiola]

Your initial question doesn't make any sense, since "mod" is an abbreviation for "modification" and not for "module". These words are not synonyms and they are not interchangeable.

I don't think you can load a kernel module from a jail (without completely compromising security guaranties, that is). No idea whether you would be able to access /dev/vboxdrv from a jail either. I doubt anybody ever tried that.

The generic kernel is modified by adding modules to create a custom kernel.

https://www.freebsd.org/doc/handbook/kernelconfig-custom-kernel.html

From inside jails, kernel modules loaded are visible. We can see this by running kldstat from within a jail. However, we cannot see if all of the directives from inside a jail's config files are applied. They can be present but fail silently.

If there's ever a question about whether or not a post makes sense, it's okay to refrain from replying. Please consider arguing less in the future. Thanks.

 

[shkhln]

The generic kernel is modified by adding modules to create a custom kernel.

https://www.freebsd.org/doc/handbook/kernelconfig-custom-kernel.html

From inside jails, kernel modules loaded are visible. We can see this by running kldstat from within a jail. However, we cannot see if all of the directives from inside a jail's config files are applied. They can be present but fail silently.

If there's ever a question about whether or not a post makes sense, it's okay to refrain from replying. Please consider arguing less in the future. Thanks.

Wtf am I reading? Learn yourself some C programming, then argue.