jails for personal purpose only or development of jail die?

[Purkuapas]

I'm trying to get to know jail technology. This is the oldest containerization technology according history page (than LXC, OpenVZ for exampel) and is positioned as much more stable than Linux-solution. Nevertheless, I found in the first hours of using jail that it can not be used in production for example, for isolation and hosting customers and despite the fact that this is the old technology - it is still not ready for use.

First: i can't run in jail two Postgres instance. Postgres or PHP APC and other stuff require shared memory. But jail do not isolate shared memory between jails - how it can be ready-to-production and mature containerization system when I can read and write from one jail to other?! How can I influence the second client will not use the same user id (recomendation from official documentation), if the customers is different users ?

second: Unable to shutdown or reboot jail inside the jail.

third: A single process id namepace.

fourth: a poor cpu limits. For example, when i limit cpu usage on OpenVZ or LXC (via cgroups) - i've really to get slow machine. But when i use RCTL - I got a working jail quickly during the X system ticks, and which sleeps for Y system ticks - if i login to jail from network and press and hold any key - i get some output then sleep, get some output and sleep Zzzz.

No loopback in jail. I setup services to bind on 127.0.0.1 only for security reason, but instead i got services which is available from external network.

And as i see in maillist - jail and vnet is still not ready to use (memory leak and panics with pf). But how it possible? VIMAGE first introduction in FreeBSD 4x ?! And still not ready?

In addition, it lacked a good tool for the jail management and no live migration support.
I have written these problems under the influence of one of the performances on youtube.com about jails where the speaker talked about the pros jail and how this system is more mature than Linux analogue and less prone to bugs. I had the impression - if it is so mature system, why am I in the first hours of use jail learned information which puts an end to the use of jail for me?

Do I understand correctly that jail development is stopped for many years and all the forces focused on bhyve ?

 

[Purkuapas]

It seems that only bhyve is good solution (with video/vnc support from boot) for hosting customers on FreeBSD. However, I do not see anything at all, what the tools (GUI) can i use to operated with bhyve. Which stuff use FreeBSD users to create and control a Linux vm and when VMs more than tens+ ?

I found some interesting information:
https://wiki.freebsd.org/bhyve?action=diff&rev1=45&rev2=46:

==== Q: What is the largest-known bhyve deployment? 
==== A: Over 1,000 nodes with over 7,000 virtual machines.

It is very interesting information, but without the appropriate proof links with the same success it was possible to add a few zeros, for more bhyve advertising ;-)
For such large systems need to be some management system at least for the minimum action. If anyone knows where there are installation over 1'000 nodes over 7'0000 vms - may be you know name of this companies and what they are using ?

 

[gzbk]

If you want to provide environments feeling like full systems where users can have full privileges and be able to restart the machine at will, then indeed jails may not be the most appropriate solution, hardware virtualization may be more appropriate.

I'm a heavy user of jails, the main real strengths of this technology as per my use case are the following ones:

• It has a negligible overhead, therefore hardware is more efficiently used and computing power is focused on handling actual clients requests, and not wasted in successive abstraction and virtualization layers,
• Thanks to this, nearly all services can be put in their own dedicated jail with no performance loss, actually a single service can even be spread among several jails for increased isolation,
• All network communications between the jails themselves and with the exterior can be finely controlled,
  
• And at last but not least, jails file systems are directly accessible from the host system in a way which is not tamperable from within the guests, allowing efficient and secure guest's files integrity checking from the host system (OSSEC, Tripwire, etc.),

For a long time, as far as I know FreeBSD jails was indeed the only free Unix system to provide such functionality.

First: i can't run in jail two Postgres instance.

Yes, you met me on my other post where I indeed uncovered SHM related issues. For this one I fully agree, shared memory and IPC handling in FreeBSD jails is really poor.

In a standard jail usage, where jails are used by the administrator to isolate different services and not to give administrative privileges to untrusted users, then there is no problem to ensure that each UID is unique system-wide. However, the fact remains that, with enough privileges, it is possible to affect a service running in one jails from another jail through its shared memory objects and in all case it is possible to established unauthorized communication channels between jails.

I sincerely hope FreeBSD has some plans to improve this, otherwise this would mean that people needing to use applications relying on shared memory and/or IPC will have to turn themselves to alternative isolation systems (and during this time Linux's user space LXC is catching up...).

second: Unable to shutdown or reboot jail inside the jail.

It is not possible out of the box, but I've seen a project somewhere in the port repository using a clever and most probably secure communication channel between the guest and the host: the host executes and certain command within the jail and this command exit code will match some configured action to be done on the host side.
Here is an example to better explain:

1. A service running on the host executes the command in the jail, the command remains in stand-by,
2. The client request a shutdown, the command will wake up and exit with, let's say, return code 3,
3. The service running on the host collect's guest command exit code, check in the configuration files, and finds that exit code 3 is associated to the jail restart command.

Return codes can be associated to any command this way, and should not offer enough surface to allow someone from within the jail to attack the host.

third: A single process id namepace.

Can you clarify he exact issue here? If you fear about potential information leakage, FreeBSD offers (also quite a unique feature in free Unixes) the possibility to have semi-random PIDs (ie. next PID will be chosen randomly in the range last PID + 1 and last PID +n where n is configurable through a sysctl value). I do not recommend fully random PIDs since PID reuse can open worse security issues.

fourth: a poor cpu limits

I have no experience on this since I never used CPU limits. Most probably the behaviour you designed can be explained as designed more for network service than interactive ones, and maybe there are way to overcome this, but I understand the issue.

No loopback in jail. I setup services to bind on 127.0.0.1 only for security reason, but instead i got services which is available from external network.

This is a non-issue to me. Set your network aliases on the lo0 interface instead of the external one, and configure PF to route incoming connections to the right jail to make selected services available to the outside.

And as i see in maillist - jail and vnet is still not ready to use

Personally I see no use for vnet. As it is now, a good set of PF rules allow to precisely design which jail can communicate with which other jail, which one can initiate connection to the outside world and which one can receive connection from the outside, and all this with all PF features.
This seems to already cover quite a wide range of uses.

In addition, it lacked a good tool for the jail management and no live migration support.

There are good tools, there are de facto standard tools, but indeed for some reason they are not part of the base system.

This should not be a problem per se (especially if you compare with Linux where there is no base system at all), but for instance I'm still crossing my finger that ezjail team (the de facto standard tool to handle jails) will finally wake up and correct deprecated sections of their product which will cease to work in its current shape with FreeBSD 11 (an issue pending for years I think). There are already alternatives, qjail being the closest one and iocage the most serious one, but they do not necessarily offer the same functionality... Integrating something in the base system would avoid such doubts.