FreeBSD Jails vs Docker - A Scientific Approach

hardworkingnewbie said:
The problem with that is that Docker relies on Linux only kernel features like name spaces (were supposed to be introduced in FreeBSD 12, did that happen?) or control groups. So in order to get it running you've got to rewrite these portions of docker to work differently, but do the same.
Click to expand...
It's only a problem if you want it to run "natively" without any Linux components, which would render most of the ecosystem useless to you - it is my impression that the porters were originally trying port not just the CLI tool but also the container runtime. Every other non-Linux platform with Docker support doesn't port the runtime, it ports the CLI tool and runs the runtime in a VM - this may be because the effort was done by the folks behind Docker...

The Docker CLI tool (i.e. sysutils/docker, which is (aside from patches) the same Docker CLI tool I have on my work Mac and my personal Linux desktop) does not require Linux only kernel features like name spaces. It just requires the configuration to talk to the Docker daemon which could be running on a local Linux VM or a remote Linux host.

A while ago I mostly got colima running on FreeBSD, it's basically some wrapper around qemu that creates a VM that runs the Docker daemon. The bit that stopped me going any further was how old sysutils/docker was and how much patching was required. If sysutils/docker was up-to-date and worked then I think those that wanted it could have Docker on FreeBSD in the same way that macOS and Windows has Docker.

And I keep saying Docker but mean to say "Linux container", because Docker is just a particular implementation these days.

Slightly off topic, but personally I'm quite excited to see where the runj project goes.
 
I am least person authorized to speak about such topic but I couldn't find on which Linux distro the research was made.

Did I miss this info?

Thanks,

F.
 
That’s the actual question, isn’t it?! 🤔 I mean we can sit here an talk the pro’s an con’s or we just try to contradict that paper. Can’t be that hard. 🤷‍♂️

Vultr offers Intel and FreeBSD for a fair price, so hardware will not be an issue.

Filesystem I would stick with UFS2.

So what’s the Docker distribution in the business, we can pick a fight with? 😆

Which settings are needed in sysctl.conf to get FreeBSD ready for the fight? 😉

An in the end we put that all in an ansible playbook to keep it documented and reproducible, an then it’s on I guess! 👀😆😉
 
Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:

BastilleBSD

Just stumbled on it by accident. 😮

Still haven’t found the time though, to actually spend some time on this. 😅
 
tanis said:
Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:

BastilleBSD

Just stumbled on it by accident. 😮

Still haven’t found the time though, to actually spend some time on this. 😅
Click to expand...
Bastille is really good, I've been using it for my homelab and my vps's to handle separation of different servers. My main gripe with it would be getting it to play nice with my pf rules but that's mostly just on me to fix; and networking while trying to keep the entire setup confined to the jails through Bastille becomes a bit of a burden. It's definatly worth a look if you have a lot of jails to deal with, and makes setting them all up with templates easy. Saves time mostly though.
 
fraxamo said:
Have you seen point #8 at this link?
Click to expand...
Yup, I've read the documentation, my issues stem from trying to redirect stuff between jails with nginx acting as a reverse web proxy. Besides that though it might be a bit till I can even tinker with the project. I bricked my server during a bios upgrade so now I'm waiting on a programmer and a bunch of windbond chips.
 
Jake0162 said:
Yup, I've read the documentation, my issues stem from trying to redirect stuff between jails with nginx acting as a reverse web proxy. Besides that though it might be a bit till I can even tinker with the project. I bricked my server during a bios upgrade so now I'm waiting on a programmer and a bunch of windbond chips.
Click to expand...
I would suggest opening a separate thread/topic about this in the corresponding category (and providing more details).
 
tanis said:
Last week someone told me they wanna start to use Docker in a enterprise environment and I justed didn't asked for further details, because my mind was off doing something else at that time and I went on with my business. Today, for whatever reason, that chat came back to me and itched me the whole day. So I started a bit researcher Docker as I'm not familiar with the technology, sure I read about it in the news and I heard people talk about it, but I use jails so I never gave it 2nd thought till today.
My google search: freebsds jail vs docker

3rd result:
Jails vs Docker - A performance comparison of different container technologies - Christian Ryding & Rickard Johansson

So I read that, well because I thought two young eager computer engineers willing to travel to uncharted land. To my surprise this paper is rather "thin", skinny comes to mind, regarding technical details.

I'm thinking about recreating that paper, BUT with a proper tweaked FreeBSD to make it a fair street fight and collect all the technical details here, just for the fun of it. 😆
Click to expand...
How / why should FreeBSD be tweaked ?
 
mer said:
sysutils/docker-machine port says "tool to create docker hosts"
A way that we've found docker useful at work is "build environments". You know how we all have preferences in what makes the perfect workstation? Well if you are working on a project with a bunch of writing software you can run into "well it built and ran fine on my system,not sure why it crashes on yours".
So create a docker container that has the build environment (should also be close to or the same as the runtime environment) that everyone uses to actually compile test in before checking into the SCM tool. Gets rid of a lot of headaches. It also can serve as the basis for git CI builds and if you do it right, also lets you create a debug environment when you have to look at core files for a released image.

Is Docker really any better or any worse than other virtualization techniques? Maybe, Maybe not. Depends on exactly what your requirements are, what is available.
Click to expand...
The issue with this approach is that it obfuscates the dependencies of your software. "I don't know how/why it worked, so I'll just package the whole system". The convenience / laziness this affords is a benefit, but this isn't an approach I'm in favor of.
 
tanis said:
Looks like I wasn’t up to date.There is actually some technology around since 2018 which aims to play in the same ballpark like Docker and it’s successor Kubernetes:

BastilleBSD

Just stumbled on it by accident. 😮

Still haven’t found the time though, to actually spend some time on this. 😅
Click to expand...
Try CBSD also. Used it in practice on a small scale and it is good.

What matters a lot with these tools is the active user base and the active contributor base. This is where jail tools fall behind.
 
I personally think the main issues with this paper come down to the amount of uncontrolled variables there are.
Two completely different operating systems, running completely different filesystems with different defaults, one might have security mitigations, etc. and you expect to be able to compare virtualisation systems, which again are virtualising two completely different operating systems.

Much better would be to take the delta in performance between the host and the guest for jails and docker on their respective setups, and compare those as percentages. That way you aren't just plotting apples and oranges and are actually comparing the relative performance of the technology.
 
jordanch said:
The issue with this approach is that it obfuscates the dependencies of your software. "I don't know how/why it worked, so I'll just package the whole system". The convenience / laziness this affords is a benefit, but this isn't an approach I'm in favor of.
Click to expand...
No it does not. It actually defines them. The bigger reason is it allows users to develop on their system of choice and ensure whatever code they write will build compile and run on the intended target. They can also test in the actual deployment environment before merging in code. Much cheaper to download a docker container than to send physical hardware around the world for developers. Say the deployment is on specific versions of a specific OS with specific versions of support packages, create a container then a developer can write code on a Mac, Windows, Linux, BSD, Atari 800, as long as there is support for docker containers. The rest of the team has a level of confidence that nobody breaks the build.
Did you gloss over the part about having it tied into git with the Continuous Integration bits? Consistent build envirionments, you don't need to find dedicated hardware with that exact set of software, you need to add a new deployment, simply create a new container and add it to the CI stuff.
If your deployment is on say small embedded devices, the company never has enough real hardware for everyone, so a container of some sort is the easiest way for everyone to contribute.
 
Hey folks, such nice points throwed over here !
Yesterday I discovered cbsd project, as well got some contact with bhyve and microbhyve, as I'm conducting some research on container orchestration in BSD platforms. BastilleBSD is a great tool, didn't try it yet, but will.

I have a good interesting in contesting this research paper presented, as evaluating containers performance, way of working, packaging and providing development environments and production systems with virtualised technologies varies in so many ways, that comparing all of that in such raw means, can lead to almost infinity misconcepted understandings as well conclusions.

As I'm entering the jails world as of now, I don't know how is the developer experience and keeping them working in production, but I would love to contribute to get this better and better. Yeah, I know won't must not follow the caddle and do everything for only the fame and the glory, but improving the way people ship code, build stuff and get more productive as well massive scale being played in background and at the corners, is a huge win for the whole ecosystem. The work behind cbsd and runj is such demonstration of all of those potential.

tanis any thoughs on how can we conduct those experiments ? I think we can start by pointing all of the misleadings done in the paper, and pointing the "right way" to get, at least, the most optimal metrics for a fair fight.
 
mer said:
sysutils/docker-machine port says "tool to create docker hosts"
A way that we've found docker useful at work is "build environments". You know how we all have preferences in what makes the perfect workstation? Well if you are working on a project with a bunch of writing software you can run into "well it built and ran fine on my system,not sure why it crashes on yours".
So create a docker container that has the build environment (should also be close to or the same as the runtime environment) that everyone uses to actually compile test in before checking into the SCM tool. Gets rid of a lot of headaches. It also can serve as the basis for git CI builds and if you do it right, also lets you create a debug environment when you have to look at core files for a released image.

Is Docker really any better or any worse than other virtualization techniques? Maybe, Maybe not. Depends on exactly what your requirements are, what is available.
Click to expand...
Regarding CI stuff and developer experience, also from the "DevOps" side, how can jails provide such "flexbility" ? I'm asking as a newcomer to jails world.
Can we show the world that jails has the same ease-of-deployment scale ?

I have a serious interest in leveraging jails to where it deserves to be : as simple as possible for managing and using in production for whatever is possible.

Just to clarify, I'm setting a small cloud provider infrastructure, for my company and offer those services to my customers. So, basically evaluating approaches and technologies. Jails, CBSD, runj and bhyve seems to be pretty solid to us, the thing now is how to offer this to people so they can ship their systems with ease (we'll provide some config style like a YAML manifest in their SCM of choice, the rest we take care off :D ).
 
Jails are awesome from a developer standpoint. The best part is, they’re lightweight, well-designed, and fully integrated into the OS.

That means, for me, I don’t need to use a jail manager. I’ve checked out Bastille, and it’s cool, but my opinions/taste differ from Bastille. That’s part of the beauty of jails - I just have a few simple scripts to set things up how I like. There‘s a learning curve, but it seems to have clicked for me and there’s a way smaller surface area than there is with Docker and its ecosystem.

As for deployment, I haven’t used them for that yet - we’re still in R&D phase of that at my company. I am currently developing a process that uses poudriere to build ZFS jail images, and uses jectl to upgrade to new versions similar to how boot environments work.

I encourage you to experiment with jails... a big bulk of the dev world is going in a different direction obviously, but in my experience popularity does not necessarily equate to effectiveness.
 
As an example, I am evaluating monitoring and alerting tools. So today I created seven or eight jails - a couple Nginx jails, and one each for the tools I wanted to try out.

I run a script `jlfs` which makes new ZFS datasets how I like them, and extracts base.txz. My default jail.conf calls a script to set up an epair, add it to the bridge, modify rc.conf in the jail to assign the IP I set as a jail var. Each item in my jail.conf is the jail name, and an IP var, that’s it. `service jail onestart mynewjail` and I’m up and running.

Thats another thing - because jails are basically just a chroot+process namespace+network stack, you can interact with their files directly from the host if you want. They’re just files on the file system, not a shared mount, or having to run remote commands or whatever. sysrc -f /path/to/jail/etc/rc.conf lets the host manage certain things on the jail, e.g. setting ifconfig.

Jails are killer.
 
It looks like that paper was focused on comparing performance. I'm not sure that it's terribly useful to anyone except a business since CPU time and memory are easy to come by. I think if you're already using FreeBSD, jails are hard to refuse. However, on something else, Docker probably makes more sense.
 
I like to think that dockers are "application virtualization", while jails/LXC (and even VMs) are "environment virtualization".

When you install a "docker", you are actually deploying an specific application that was placed in that image; when you create a jail/LXC, you install whatever you want.

Once you understand and accept this distinction, choosing one or the other becomes easier and more natural.

And me? I use both ;-)
 
Very cool. Jails sequester a portion of the (host's) kernel such that its memory isn't shared between the host and it, in addition to the filesystem chrooting. I think I read somewhere that dockers are often used to ensure library compatibility. Though, I'm not sure what/how/if it offers memory isolation.
 
You must log in or register to reply here.
Back
Top