general/other VMs or jail? And hierarchical jails?

[DaLynX]

Hi,

I am building a new server to migrate my current setup to newer hardware.
Old setup was a linux proxmox based hypervisor with 2 linux and 2 freebsd VMs.

I am building the new one as a freebsd hypervisor. The 2 linux VMs will stay as is. What I am wondering is what would be the best way to deal with the FreeBSD ones.

Services in the FreeBSD VMs are set up in jails. Which means I could easily migrate them to jails on the new host directly, without resorting to virtualization. Should I?

Here's what I have though of so far:

Pros of jails vs VM
- more lightweight, less performance and storage overhead
- better integrated with the host (esp. ZFS) so better performance

Cons
- Isolation is not as good. If I crash the kernel, like I did recently playing with unionfs, I crash the whole host, including the linux VMs.
- That's also a concern security-wise, given some services are Internet-facing.
- Services from both VMs would be mixed on the same management level, although they do not belong to the same users / management. (I manage both but some other admins also manage one VM, but not the other where it's only me.)

A palliative for last point could be the use of hierarchical jails. But that means giving up on fine devfs filtering in service jails (unsupported).

What do you think? How would you do it and why?

 

[sko]

I could easily migrate them to jails on the new host directly, without resorting to virtualization. Should I?

Yes. Always keep the complexity and overhead at a minimum, this will benefit you greatly when upgrading, migrating or recovering from backups in case of fatal hardware failures.

- Isolation is not as good. If I crash the kernel, like I did recently playing with unionfs, I crash the whole host, including the linux VMs.

For experiments with beta software that might kill the whole system (and/or pose a risk to valuable data) one should always use a test system anyways. Any cheap used mini-PC or retired system is sufficient for such things. Even a VM could potentially drag down the host if "crashed the right way".

- That's also a concern security-wise, given some services are Internet-facing.

isolation in terms of security is pretty much the same for jails vs VMs. Anything that connects or is accessible from outside belongs in a dedicated VLAN (e.g. DMZ) anyways, which isn't routed (or very restricted) to other local networks.

- Services from both VMs would be mixed on the same management level, although they do not belong to the same users / management. (I manage both but some other admins also manage one VM, but not the other where it's only me.)

Not sure what you mean with "management level", but if you don't want the services to be on the same network, again: use different VLANs for different networks. Simply using different prefixes/subnets on the same physical network is extremely bad practice and causes all sorts of headaches.

A palliative for last point could be the use of hierarchical jails. But that means giving up on fine devfs filtering in service jails (unsupported).

I've played around with nested jails - it's entirely possible and even relatively straightforward, but I could never really justify the higher complexity of such a setup. "Jails Mastery" by Michael W. Lucas contains a whole Chapter on hierarchical jails and is overall a very thorough and deep-diving source of information regarding jails. I can only recommend it.

 

[DaLynX]

isolation in terms of security is pretty much the same for jails vs VMs. Anything that connects or is accessible from outside belongs in a dedicated VLAN (e.g. DMZ) anyways, which isn't routed (or very restricted) to other local networks.

You are right from the network point of view but I was thinking about system privilege escalation. I consider it easier to break out of a jail, for which you just need a kernel vulnerability, than out of a vm, for which you need to find a vulnerability in the virtualization layer. Also voluntary resource exhaustion would easily harm the system from a jail - not so much from a VM that only has N vCPU and X GB RAM.

How about file handles, for instance? Can you limit each jail's usage or does exhaustion mean exhaustion for the whole system?

Not sure what you mean with "management level", but if you don't want the services to be on the same network, again: use different VLANs for different networks. Simply using different prefixes/subnets on the same physical network is extremely bad practice and causes all sorts of headaches.

I meant they are not managed by the same people.

In a nutshell I currently have a shared services VM and a personal VM. Other admins do also manage the shared VM. But I am the only one managing mine.

If all jails, shared and mine, are side by side on the same host, it's easy for others to interfere with them or look inside.

I admit that's not a really strong argument since other admins are also root on the host anyway so they could enter the vm through the hypervisor if they wanted. But it's no as straightforward as entering a jail.

But anyway, I think you made your most important point here:

Always keep the complexity and overhead at a minimum, this will benefit you greatly when upgrading, migrating or recovering from backups in case of fatal hardware failures.

I trust my co-admins and complexity and overhead are my most likely ennemies here. So the simpler the better. I'm going with jails.

Thanks a lot for pitching in.

 

[patmaddox]

I prefer jails over VMs because they're just a file system. You can mount a VM image once it's been shut down (or copy the image file and mount it) but I like being able to inspect the running jail from the host.

Linux of course goes in a VM so it can't hurt anyone.

 

[DaLynX]

That's a good point. I took the habit too of doing things in jails through the host. Looking at logs, changing config, and then just exec the service restart or whatever command instead of actually entering the jail. It's just so easy.

 

[gotnull]

For experiments with beta software that might kill the whole system (and/or pose a risk to valuable data) one should always use a test system anyways. Any cheap used mini-PC or retired system is sufficient for such things. Even a VM could potentially drag down the host if "crashed the right way".

Interesting topic, I am kind of surprised by your saying, could you explain what type of events/actions can put down a VM and the host at the same time? thank you.