FreeBSD Jails vs Docker - A Scientific Approach

[rootbert]

Docker engine & containerd do not use Linux containers.
What they do is unclear to me.
They call it container for only one application. But what are the technical detail of the implementation?

Beginners Track - Is Docker technology the same as traditional Linux containers?

collabnix DockerLab Docker - Beginners Track

dockerlabs.collabnix.com

Can different docker applications talk to eachother through rpc,named-pipes,sockets. Or do they isolate & how.

they both provide the environment for running Linux containers (OCI standard). Docker is generally not used any more, people use containerd or one of the other engines nowadays (e.g. podman or cri-o etc.), depending on what they need. Those engines use two technologies of the Linux kernel: cgroups and namespaces. So cgroups are basically the technology to hierarchically organize the processes (e.g. while according to the microservices dogma there should be exactly one process per container you can definitely have much more, so all the processes of your container are together in a process group which will then be administered via the container engine).

Namespaces are, well, namespaces, and there are various sorts. Containers within the same network namespace can communicate via network. You can enable various types of communication (named-pipes, sockets) though in general you use network protocols. Containers are often spread out over many hosts and using technologies like vxlan they talk to each other transparently and do not realize that they are on other hosts.

Isolation can be quite fine-grained, you can isolate them so that no container can in any way access other containers on the same host. You can even use the virtualization technology of your cpu and use an absolute minimal kernel and actually have virtual machines that are handled like containers - this is used if you have security concerns. It is quite amazing since those extremely lightwight containers start within half a second, and on our hosts we can start > 500 of such VMs in less than 5 seconds.

 

[ct85711]

One thing if you are going to compare the 2, you need to take into account more of a generalized lifespan of the 2. To me, Docker/containerd/etc tends to encourage more of a short to medium lifespan, to where jails tend to encourage more of a medium to long term life. Now of course, there are always the outlying cases on both, depending on use cases. An anology that tends to fit well, is more of Docker being like a paper towel, while Jails are like a dish towel. Paper towels, you use and throw it away once you are done and get a "new" one when you need to use it again. Where Jails once you are done, you clean it and use the same towel again. Applying it back to the actual items, consider what happens when updating something. Jails, you simply just update the package and keep on rolling, Docker is more of a process that updates are more of you throw away the old container and make a new one on any change(s). So in the end, both can do similar jobs; it doesn't necessarily mean it is the correct tool for the job.

 

[Alain De Vos]

So if i understand you correctly you can run a "pulse-audio-server" in containerd/docker which automatic restart after it crashes.

 

[ct85711]

The main thing for Docker, is that the container's content is effectively frozen; so all changes done when it is running done like in memory (lost on restart). That is why if you need to keep the changes, you have to recreate the container (of course there is ways, to save stuff; which is equivelent to sharing files to a jail).

A prime example would be a DB service, Jails would be a better tool for having the primary DB (so it can save any changes), where as a DB in a Docker container would be good for a testing/dev environment where you don't care if you screw up as you can restart and have everything reverted to a set condition.

 

[rootbert]

So if i understand you correctly you can run a "pulse-audio-server" in containerd/docker which automatic restart after it crashes.

yes and no: if the software crashes in the sense of the process stops or is killed, then (in the default config) it restarts automatically. However, if the process just hangs and is in an unknown state where the process keeps running then of course you have to kill the container manually.

The main thing for Docker, is that the container's content is effectively frozen; so all changes done when it is running done like in memory (lost on restart). That is why if you need to keep the changes, you have to recreate the container (of course there is ways, to save stuff; which is equivelent to sharing files to a jail).

A prime example would be a DB service, Jails would be a better tool for having the primary DB (so it can save any changes), where as a DB in a Docker container would be good for a testing/dev environment where you don't care if you screw up as you can restart and have everything reverted to a set condition.

There is something to add: if you want to keep the data you mount a volume into the container, like you would do with mount_nullfs under FreeBSD. So this is also ideal for running a primary DB: just stop your postgres 13.1 container which uses /var/lib/postgres mounted from the host, snapshot the hosts data directory, and start postgres 13.2 with /var/lib/postgres coming from the host. Problem? yes: stop container, restore snapshot and start postgres 13.1 again with /var/lib/postgres coming from the host.

So basically, I would say jails are more admin-centric with focus on snowflake systems, like a single asterisk/samba/ftp whatever server, equivalent to what openvz in Linux was or currently LXC/LXD is. OCI containers are more for delivering applications, like create an application, package it into an immutable container, distribute it to your clients servers or roll it out on your thousand hosts within the blink of an eye. Both can be used for both requirements, however, the toolset in the Linux OCI ecosystem is just tailored for this kind of deployment, with jails it is a lot of work because you basically have to write all the tools around that yourself.

 

[Jake0162]

ct85711 So is it pretty much like running a jail set with a kernel security level that makes it immutable?

With flags set I guess you could load a DB into /tmp on the jails boot and then work on that for a test/dev environment. After the jail rebooted you would be back to a fresh environment. But there is probably better ways to handle it than what I suggested.

I haven't spent a whole lot of time using Docker but I'm kind of interested in how it works now.

 

[ct85711]

So is it pretty much like running a jail set with a kernel security level that makes it immutable?

In short yes, both jails and docker does the same job in the end, with a different approach. Outside of Docker/linux containers having an advantage in being designed so you can easily distribute the container onto another system with only "cost" being to downloading the already configured container on that system. Jails from what I have seen, have the advantage of being rock stable while Docker is good for speed(quick to setup a new container)/ease to configure.

 

[rootbert]

ct85711 So is it pretty much like running a jail set with a kernel security level that makes it immutable?

With flags set I guess you could load a DB into /tmp on the jails boot and then work on that for a test/dev environment. After the jail rebooted you would be back to a fresh environment. But there is probably better ways to handle it than what I suggested.

I haven't spent a whole lot of time using Docker but I'm kind of interested in how it works now.

I will refer to OCI container since docker is the wrong term, docker is dead. Per default they are not immutable, but you can configure the container to be immutable. Consider throw-away-binary: you start the container, you change stuff inside the container, you stop it. When you start it again it is in the same state as before any change - you always have the same state when the container starts. How does one save the changes? - data is usually mounted into the container where you need it, e.g. the hosts /myzpool/postgres goes into container "postgres:12.3" to /var/lib/postgres - so you can persist. Configuration is either passed via environment variables or distributed inside the container, or via mounting e.g. host:/myzpool/nginx-configs/srv01/nginx.conf -> nginx-appcontainer1:/etc/nginx/nginx.conf.


Luckily the whole OCI ecosystem has stabilized since the standardization and is equally stable to jails, but of course you need to consider all the updates on the Linux kernel and systemd to reboot. You have lots of options to choose and really cool technologies, it is a vast universe on its own. Want more security? - use immutable containers, just allow one binary to be executed inside the container, disable unnecessary systemcalls or even run the container inside a tiny kernel not implementing potentially dangerous syscalls or even take advantage of hardware virtualization an run the container with its own trimmed down Linux kernel.

 

[6502]

I read the comparison results (Jails vs Docker) but it seems they are done only with ZFS. Are there similar tests with UFS for FreeBSD?

 

[vermaden]

Last week someone told me they wanna start to use Docker in a enterprise environment and I justed didn't asked for further details, because my mind was off doing something else at that time and I went on with my business. Today, for whatever reason, that chat came back to me and itched me the whole day. So I started a bit researcher Docker as I'm not familiar with the technology, sure I read about it in the news and I heard people talk about it, but I use jails so I never gave it 2nd thought till today.
My google search: freebsds jail vs docker

3rd result:
Jails vs Docker - A performance comparison of different container technologies - Christian Ryding & Rickard Johansson

So I read that, well because I thought two young eager computer engineers willing to travel to uncharted land. To my surprise this paper is rather "thin", skinny comes to mind, regarding technical details.

I'm thinking about recreating that paper, BUT with a proper tweaked FreeBSD to make it a fair street fight and collect all the technical details here, just for the fun of it.

The I/O test should be done on ZFS for both technologies ... or also include UFS tests on FreeBSD.

With current test EXT4 on Linux and ZFS on FreeBSD its comparing performance of filesystems - not the container technologies.

 

[k3y5]

I will refer to OCI container since docker is the wrong term, docker is dead. Per default they are not immutable, but you can configure the container to be immutable. Consider throw-away-binary: you start the container, you change stuff inside the container, you stop it. When you start it again it is in the same state as before any change - you always have the same state when the container starts. How does one save the changes? - data is usually mounted into the container where you need it, e.g. the hosts /myzpool/postgres goes into container "postgres:12.3" to /var/lib/postgres - so you can persist. Configuration is either passed via environment variables or distributed inside the container, or via mounting e.g. host:/myzpool/nginx-configs/srv01/nginx.conf -> nginx-appcontainer1:/etc/nginx/nginx.conf.


Luckily the whole OCI ecosystem has stabilized since the standardization and is equally stable to jails, but of course you need to consider all the updates on the Linux kernel and systemd to reboot. You have lots of options to choose and really cool technologies, it is a vast universe on its own. Want more security? - use immutable containers, just allow one binary to be executed inside the container, disable unnecessary systemcalls or even run the container inside a tiny kernel not implementing potentially dangerous syscalls or even take advantage of hardware virtualization an run the container with its own trimmed down Linux kernel.

OCI containers have active participants submitting code from within the Chinese government. Are you worried about system vulnerabilities built into the code base? It has been their modus operandi for most of the projects they've actively pursued.

The thing the urks me is the closed door security policy. You have to jump through hoops to get access to the mailing list for security issues. In my opinion those should be public by default.

 

[rootbert]

OCI containers have active participants submitting code from within the Chinese government. Are you worried about system vulnerabilities built into the code base? It has been their modus operandi for most of the projects they've actively pursued.

well this has nothing to do with OCI per se, it's a "problem" of open source in general.