general/other Linux Veteran Wanting to Convert to FreeBSD

[Shake&Bake]

Hello everyone. This is my first post in FreeBSD forums as a Linux user of 3 years. My question stems to using FreeBSD using K8s/K3s or Docker in general. I have read documentation on jails which has piqued my interest. However, I have not found much documentation on utilizing the services. Also, I have questions in regards to the compatibility of other services when using FreeBSD.

My questions have to deal mostly with containerization using K8s/K3s. I have noticed the following:
- Many FreeBSD evangelicals who use FreeBSD have a disdain for Docker (which I can understand since we might as well use nerdctl). Is jails a substitute for Docker, or would nerdctl be a better option?
- Searching the web, I don't see much implementation of K8s/K3s homelabs with FreeBSD. Is K8s/K3s difficult to setup on FreeBSD? (I wrote an Ansible script to setup K3s on my homelab with Debian Bullseye but when I look online, I cannot find any references for setting K3s up on my servers I have except using bhyve which I do not want to virtualize a Linux image. There is a reason why I want to move away from Linux.)
- As for adding projects such as Jellyfin, Wazuh, Calibre, etc, once containerized will these cause any issues on FreeBSD? Random question I know but just curious.

Thanks for the help in answering my generalized questions! Complete noob to this world and would love to learn more in why FreeBSD is used with companies like Netflix and Juniper.

 

[smithi]

I don't know, but think your question deserves a bump.

 

[Jose]

Hello everyone. This is my first post in FreeBSD forums as a Linux user of 3 years. My question stems to using FreeBSD using K8s/K3s or Docker in general. I have read documentation on jails which has piqued my interest. However, I have not found much documentation on utilizing the services. Also, I have questions in regards to the compatibility of other services when using FreeBSD.

Welcome! What services do you mean? If you mean Kubernetes, no, that will not work on Freebsd because it's built on top of Docker.

My questions have to deal mostly with containerization using K8s/K3s. I have noticed the following:
- Many FreeBSD evangelicals who use FreeBSD have a disdain for Docker (which I can understand since we might as well use nerdctl). Is jails a substitute for Docker, or would nerdctl be a better option?

Jails were implemented before Docker was a thing. I consider Docker an inferior implementation of the same idea. Way too complicated, and the fact networking was an afterthought is distressing.

- Searching the web, I don't see much implementation of K8s/K3s homelabs with FreeBSD. Is K8s/K3s difficult to setup on FreeBSD? (I wrote an Ansible script to setup K3s on my homelab with Debian Bullseye but when I look online, I cannot find any references for setting K3s up on my servers I have except using bhyve which I do not want to virtualize a Linux image. There is a reason why I want to move away from Linux.)

Yeah, and you won't because those technologies don't work on Freebsd. See above.

- As for adding projects such as Jellyfin, Wazuh, Calibre, etc, once containerized will these cause any issues on FreeBSD? Random question I know but just curious.

No idea. Start with, "do they depend on Docker?" If they answer is "yes" then they will not work on Freebsd. Otherwise start a thread on each and maybe there'll be forum members with specific experience.

Thanks for the help in answering my generalized questions! Complete noob to this world and would love to learn more in why FreeBSD is used with companies like Netflix and Juniper.

Not to discourage you, but it sounds like you're looking for technical reasons. There are small ways in which Frebsd is somewhat better than Linux, but you should know that the BSD license was probably a big part of those companies' decision to use Freebsd.

 

[YYY__]

I see it in another way.
Docker is a virtualization tool with built-in Linux kernel functionality and strong commercial support behind it.
FreeBSD is a virtualization tool built into the FreeBSD kernel.
Both depend on the underlying kernel. Whether one or the other is better is up to you.

Thanks for the help in answering my generalized questions! Complete noob to this world and would love to learn more in why FreeBSD is used with companies like Netflix and Juniper.

System coherence, network capabilities, KISS principle and BSD license.

 

[sidetone]

There's sysutils/docker in Ports.

sysutils/bastille and other jails are developed on FreeBSD or are native on BSD's.

For Comparison TrueNAS, which for most of its existence was solely a FreeBSD derivative, and it still has a version which is a FreeBSD derivative, TrueNAS has Linux versions which uses Docker. The BSD version of TrueNAS uses FreeBSD tools.

The simplest way is through FreeBSD's native tools. If Docker is to be used, extended parts that depend on it should be in their own repository or ports tree, because that keeps things clear and keeps things from being unnecessarily complicated in ports.

 

[YYY__]

How does this Docker work on FreeBSD, does it need Linuxulator?
...

Docker - FreeBSD Wiki

 

[GoNeFast_01]

As for adding projects such as Jellyfin, Wazuh, Calibre, etc, once containerized will these cause any issues on FreeBSD? Random question I know but just curious.

I'll tell you my experience...

I do not want to virtualize a Linux image

Get ready to use a bhyve, or qemu nothing else to 100% replace linux base apps in FreeBSD in some instances...

You can try a linuxulator/chroot like (BUT you can see the last message on the post below for Brave I had error getting it to work on 13.1p2)

[Linuxulator] How to install Brave (Linux app) on FreeBSD 13.0+

Thanks for your efforts on this. Everything works fine, netflix etc fantastic. However I have one issue. I have a titan key setup with my google account so when I try and login to google it tries to probe udev, which causes a segfault: ERROR:bus.cc(393)] Failed to connect to the bus: Did not...

forums.freebsd.org

Getting these native linux apps to run even on a chroot environment will require substantial technical knowledge and compiling kernels, editing / rewriting libraries in C, etc...

Anyways why would you want a Linux Virtualized on FBSD instead of Linux simple ZFS, plus all of these tools have an alternative or found their roots in a BSD alternative app first, you just need to find a new path forward.

 

[hruodr]

Dost thou renounce the Daemon and all his works? Then remain with the Penguin!

 

[LibreQuest]

Why do you want to move away from Linux?

 

[GoNeFast_01]

To add some interesting comical quotes from Reddit and for some suggestion to your path forward.
Why does the freebsd community hate docker?

I hate them because it’s ALL insanely overly complex garbage software.
Software has gotten into such a dumpster fire state these days. I don’t think the people writing modern software have any idea how to write software to be used by normal people.
You can argue until you are blue in the face the software in questions solves a problem.
To me it doesn’t matter, if I have to read entire manuals, google, debug, spend countless man hours of weeks to install, config and support it it’s dead to me. D E A D. I’ve got better things to do.
No one seems to care. Modern software just gets more massive, convoluted, further and further away from the Unix philosophy, and the raging dumpster fire burns hotter and brighter. And no one seems to care. No one says NO!
So there we are. IT has become such a nuclear trash fire. This is the world we live in.

All virtual systems (Windows, Linux and FreeBSD) are running on bhyve and jails.So far I cannot complain since I have a script that just takes a few parameters and creates machines while allocating resources to them.
As for services getting restarted, necessary updates, e.t.c all OS's have that kind of thing built into them and I do not see the need for extra complexity where the OS (host or guest) can just handle that for you.
The overheads are minimal, migrations can be done live, redundancy is easy to achieve and most importantly stability is assured as nothing will change overnight.
That beautiful stability and the absolute "only when necessary" culture is why I have not only stuck with BSD and tell all my friends about it but do my best to offer solutions to their problems based on it!

I think for a large part FreeBSD users have seen some things, and aren't going to jump onto the latest bandwagon shiny. There are countless "IT guys" out there that think because they can type 'docker run' that they're sysadmins.
It's a bit of BOFH, and it's proper to be so.

Because everything about their designs is phoned in.
While we don't have a single tool for it, we have multiple small tools that serve as the building block to do the same thing, but better and easier to manage.
Also would not say they are the future, just the flavor of the day, which when you look at the history of IT tends to usually be dumb as fuck and then burn out given it has no staying power.
Between etcupdate, pkg, jail, pf/ipfw, and rctl we have a damn powerful system once you start to use Rex or Ansible. Especially if you choose to use Rex(makes doing lots of stuff that is hard in Ansible trivial).

Disclaimer: I am linux user.
Docker is not portable. Its only native to Linux, other OS will need to run it under thin VM. Docker is wasting more resources, disk, cpu, bandwidth, then what people though.
Kubernetes is the great marketing software ever. Instead of focusing on logging and monitoring, we wasted our hours on another layer of software. What a great tragedy!

I rarely comment on Reddit, but man did I read some BS rn. I came expecting everyone here promoting a better alternative and explaining that jails and docker serve different purposes, but it's just IT boomers parroting each other and saying "X technology is bad because it's popular" (Just in case it wasn't not clear, my tone was hostile lol).
Docker != containers. Docker is a tool to provision the containers. Whatever goes on behind the scene behind the scene with linux namespaces, runc, containerd, and all of that doesn't matter from the user's point of view who just wants to write a Dockerfile that manages dependencies configures whatever service or application is, Jails simply don't provide that.
Jails are more lightweight than linux containers. They're way older and mature (implying stability and a little more security). In fact, FreeBSD does a lot of things in nicer ways (i.e. zfs, nicer security approach, etc), and there actually are solutions to somewhat of a replacement for docker/podman for FreeBSD.
Have a look at https://www.freebsd.org/status/report-2021-04-2021-06/pot/ which provides a container based approach to jails. K8s... is another story, and I don't know of any solutions other than nomad (watch https://archive.fosdem.org/2020/schedule/event/orchestrating_jails/) and Nomad itself is nowhere near as mature as Kubernetes.
Back to the rant; I've seen this type of shit in C/C++ communities where everyone hates Rust, or when old lisp programmers are told machine learning is a thing. Yes, new technology can be bad, and it will not be mature, but have a bit of an open mind, and maybe realize that they try to solve a problem that the older tools did not, and maybe not repel newbies like yourself and I from using FreeBSD.

Linux-only (but BSD-curious) guy here. Let's talk about Docker specifically.
Docker was first to market on wrapping up a lot of features. It was a really neat trick that brought a lot of nifty tech that only server admins understood down to your developer's laptop. Go back and watch the original PyCon docker lightning talk.
But Docker the company was poorly managed, and the software they produced, while it saw a lot of initial adoption, is now being factored out of the equation in Kubernetes, and in all my servers.
The other early competitor was CoreOS, who actually made all the RIGHT choices, in my opinion. Rkt was a better container design. CoreOS gave us etcd which is basically the reason K8s works.
Docker gave us...Dockerfiles? No thanks.
Everything about Docker is a lowest-common-denominator approach. If a container is supposed to be the next gen of packaging for your software, I feel like they should do a little bit more for us than being a tarball of tarballs with a raw string for versioning that you have to parse and figure out yourself. Anyone who understands linux packaging knows that containers don't REALLY solve the problem, they're just a tarball built in a chroot. And they rely on debian and red hat and archlinux ANYWAY for the hard work of putting together a working distribution.
As for Kubernetes, it actually has some really neat stuff, but 95% of companies adopt it prematurely. Kubernetes is a technology _for building platforms_. That's why tons of vendors have sprung up around it, because they can add their little bit of special sauce and capture some revenue. Kubernetes is like Java EE or Windows Server. And honestly that's not the kind of stuff I like to use

Take your pick I am agnostic honestly to the Docker/ Kubernetes debate, I use what I need and try to always learn the new "IT cool thing" cause I talk to different management levels/IT abstractions but the quick fix is just emulation/bhyve call it a day or find new path forward for your use case.

There are Jail wrappers and Bhyve wrappers that offer a lot of the functionality of Docker but I've learn to do everything manually just in case those tools go down/bug.

POT

GitHub - bsdpot/pot: pot: another container framework for FreeBSD, based on jails, ZFS and pf

pot: another container framework for FreeBSD, based on jails, ZFS and pf - bsdpot/pot

github.com

BASTILLE

GitHub - BastilleBSD/bastille: Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD. - BastilleBSD/bastille

github.com

CBSD

GitHub - cbsd/cbsd: Yet one more wrapper around jail, bhyve, QEMU and XEN

Yet one more wrapper around jail, bhyve, QEMU and XEN - cbsd/cbsd

github.com

APPJAIL

GitHub - DtxdF/AppJail: Simple and easy-to-use tool for creating portable jails.

Simple and easy-to-use tool for creating portable jails. - DtxdF/AppJail

github.com

IOCAGE

GitHub - iocage/iocage: A FreeBSD jail manager written in Python 3

A FreeBSD jail manager written in Python 3. Contribute to iocage/iocage development by creating an account on GitHub.

github.com

VM-BHYVE = Churchers Bhyve

GitHub - churchers/vm-bhyve: Shell based, minimal dependency bhyve manager

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.

github.com

EZJAIL

ezjail-admin

man.freebsd.org

I've use all above or play with them... But when something goes , I ALWAYS revert back to old favorite jail && bhyve + tmux.

Base on what I saw the closest thing to https://github.com/containerd/nerdctl or https://www.freshports.org/sysutils/containerd will be Nomad https://github.com/hashicorp/nomad BUT beware of the latest drama with HashiCorp license changing to BSL

 

[Shake&Bake]

Thank you all for the outpouring support with all the information everyone! This really helps me out in reading up and seeing different purposes of Linux and FreeBSD. And yes, I feel the uneasiness of Docker being oh so clunky.

I do have one last question in the use of FreeBSD: Is there documentation on what would be the shining definition of using FreeBSD compared to Linux when it comes to a server? For instance, more and more companies are transitioning towards micro-services with K8s; however, like what was said, networking and storage capabilities outshine Linux when it comes to performance and security with FreeBSD. Am I understanding this correctly?

 

[Shake&Bake]

Why do you want to move away from Linux?

Main reason for me is the added security and getting into the world of FreeBSD. I heard big companies use it but never knew why. Many articles I have read on my spare time delve into, for example, Netflix pushing petabytes of streaming data to users around the world in how they have configured their own hardware that supports their in-house projects. It just fascinates me how they accomplish this

 

[Beastie7]

- Many FreeBSD evangelicals who use FreeBSD have a disdain for Docker (which I can understand since we might as well use nerdctl). Is jails a substitute for Docker, or would nerdctl be a better option?

The concept behind Docker was more or less inspired by Jails. Secure process partitioning was first conceived with Jails. It's practically chroot on steroids (another BSD invention). I wouldn't call Jails a substitute for anything. It's the security feature in this regard; all else came after that.

I don't think folks here hate Docker per-se, but more the unreasonable hype of what it isn't, and it being heavily tied to Linux dependencies. There is no security in Docker, period.

- Searching the web, I don't see much implementation of K8s/K3s homelabs with FreeBSD. Is K8s/K3s difficult to setup on FreeBSD? (I wrote an Ansible script to setup K3s on my homelab with Debian Bullseye but when I look online, I cannot find any references for setting K3s up on my servers I have except using bhyve which I do not want to virtualize a Linux image. There is a reason why I want to move away from Linux.)

Kubernetes has way too many Linux dependencies. It simply won't work.

- As for adding projects such as Jellyfin, Wazuh, Calibre, etc, once containerized will these cause any issues on FreeBSD? Random question I know but just curious.

That's a matter of upstream providing a FreeBSD port, or if someone in the community cares enough to port to FreeBSD themselves. Perhaps, you?

 

[Adamo]

Jails are a similar system to docker, and are directly supported by the FreeBSD operating system. I've tried a number of orchestration system for jails, and there's nothing yet that reliably handles the same level of orchestration through automation as Docker does, especially as it relates to networking, which you pretty much need to set up yourself in FreeBSD. But I use jails for the same PURPOSE: I've written up my own configurations to support it. It was more work up front than "docker run blah" but the end result is rock solid, just like most of FreeBSD.

I successfully setup Docker using bhyve, but I never used it for anything. I may end up trying to support linux containerization using that approach, but so far I've used jails instead. If I really needed linux containerization right now in my personal data center, I'd probably use a separate server with a linux OS.

If your monitoring software supports FreeBSD then I wouldn't imagine they would "cause problems". FreeBSD is the most reliable, main-stream OS available -- it's hard to break (not impossible!). If you want to run them in containers, and they need access to particular system resources that you wouldn't normally grant to a default jail/container, then read the man pages and understand the security and reliability implications for adding that access: it is certainly possible to add access to the resources you need, and this is controlled through a number of parameters, whose default options can often be appended or tweaked.

 

[jecxjo]

For some added history, the linux world started out with lxc (Linux Containers) which is an OS Level virtualization system that was based off of how Jails works. Using cgroups (kernel level control of resource allocation for processes) lxc can run a single kernel with multiple systems under it where the host OS has direct visibility and control of the virtualized processes. Or in simple terms and beefed up chroot.

Originally Docker was a wrapper around lxc. Think it was around v1.10 this support was dropped for libcontainer which is their own version of a controller for the same functionality.

 

[hardworkingnewbie]

I do have one last question in the use of FreeBSD: Is there documentation on what would be the shining definition of using FreeBSD compared to Linux when it comes to a server? For instance, more and more companies are transitioning towards micro-services with K8s; however, like what was said, networking and storage capabilities outshine Linux when it comes to performance and security with FreeBSD. Am I understanding this correctly?

Well regarding security: FreeBSD is as secure as Linux, and vice versa. There have been enough analyses done by security experts on both, and their mutual agreement always was that while Linux has more CVEs in the same time compared to FreeBSD is that FreeBSD is used less, therefore less eye balls on it. But when looking at code quality and the kernel, it's easy to find enough bugs and holes in FreeBSD as well.

If you really want an OS where security is the top priority you have to go with OpenBSD, which has its own quirks and caveats as well.

Having said that, FreeBSD is easier to understand in certain areas, like init system, compared to Linux with systemd nowadays almost everywhere. And therefore more predictable in its behaviour as well, and its core tools feel more consistent to me as well compared to Linux.

On network performance: there is a reason why ftp.cdrom.com ran for years with FreeBSD, topping concurrency and bandwidth records frequently in a row - back then Linux's network stack was way worse compared to FreeBSD than it is nowadays. There's also a reason why Netflix uses FreeBSD for its streaming boxes. There is a reason why most high volume web sites ran FreeBSD for a long time according to Netcraft's web server survey.

There's a reason why in 2014 Facebook was looking for a linux kernel developer to improve its network stack to rival or exceed that of FreeBSD (https://www.theregister.com/2014/08/07/facebook_wants_linux_networking_as_good_as_freebsd/).

But this is both at the high end range, and also let's not forget that Youtube runs on Linux. Having said that, in terms of private/SOHO networks the performance difference, if any, is negligable between both OSes. Actually Linux has there one main advantage though still at the moment, it has better support for modern wifi standards. FreeBSD, while trying to catch up, is here still lagging behind Linux.

 

[Jose]

I do have one last question in the use of FreeBSD: Is there documentation on what would be the shining definition of using FreeBSD compared to Linux when it comes to a server? For instance, more and more companies are transitioning towards micro-services with K8s...

That is certainly the way the herd is running right now. Freebsd is probably not the right choice if you're looking to do things the same way as everyone else.

This is more the vibe for the BSDs

Minor Threat - Out of Step (010)

These songs are also available on the Complete Discography Maxi CD.

dischord.com

 

[sidetone]

TrueNAS, when it was solely based on FreeBSD, has been used by the US NAVY and other governmental organizations. The list has changed, since Redhat became a client, and TrueNAS offered services for Linux Docker, which now includes other US government organizations. That entity is no longer on that list, but is replaced by other US military branches: not sure if its no longer used by them, or if is ommitted due to it being a curated list. There was money on the table, so iXsystems implemented an inferior product, because people want Linux, and likely because Redhat got involved perhaps along a few other organizations. RedHat is a client that wasn't on that list, when it was FreeNAS.

I wish there were a press release or case study (published as opensource) of the clients that TrueNAS had when it was FreeNAS and based solely on FreeBSD. And further publications on which companies use the BSD version of TrueNAS. FreeNAS was used by highly reputable organizations and government branches for the longest time, which lended to its reputation, and TrueNAS only offered Linux services most recently. Hopefully, there are archived screenshots of clients when it was FreeNAS which would be helpful. Better yet, a year by year list of their clients, intended to show it as FreeNAS. It's in the interest of FreeBSD and other BSD's to have old client lists of FreeNAS preserved.

I don't like that part, because it waters down the reputation of BSD, and lends reputation to Linux, when it had nothing to do with FreeNAS until it became TrueNAS most recently. I'm for iXsystems making money and using their choice. I wish they kept a clear separation of clients which used FreeNAS, and also now use the current TrueNAS version that's based on BSD. Also, companies produce better products when they specialize and don't branch out too far to multiple types of products, which would be in this case having branches of Linux and BSD. It will work for now, but companies don't provide the best for long when that happens.


However, on the front page of the FreeBSD website, it shows FreeBSD clients, including Netflix.

 

[timotheosh]

Hello, and welcome to the FreeBSD community!. I do hope you realize that you joined a tech community, not a cult. Many of us use Linux, Mac and Windows. Monogomous relationships are meant between people, not technologies.

Cloud nerd here. I use Linux professionally, and did work with FreeBSD professionally a decade ago or so, when I worked at NetApp.
I don't see any of the fore-mentioned companies abandoning Linux for FreeBSD. What you see are unique technical needs where the standard methods of operation do not solve the problem at hand. There are cases where FreeBSD fills a need better than Linux or even any other competing OS.
In the case of NetApp, I never did find a whitepaper or set of slides as to why they chose FreeBSD as a base for their OnTap operating system, but I suspect the licensing played a major role. In the case for Netflix, there was a very specific complex problem with video delivery that involved overcoming latencies between kernel and userspace. A combination of AMD EPYC CPU technology and FreeBSD foot the bill. The solution greatly increases latency between outside nodes, so it isn't a "cure all" solution, but a very specific technical solution. More on that here https://fliphtml5.com/uowif/wfka/basic
NetFlix and NetApp still are widely using Linux servers for just about everything else. Maybe that has more to do with industry standardization where "good enough" reigns.

There are plenty of interesting projects people are working on that involve FreeBSD and the Cloud.
Kubernetes, docker, and containerization in general are not the best platforms in terms of security. We did a hackathon six years ago where I work, where we investigated the use of Unikernels in AWS. This unikernel, does not advertise it, but they are using a FreeBSD kernel for their Unikernel. https://osv.io/

MicroVM's and Unikernels could end up displacing containerization. And FreeBSD has a voice in this world.

AWS' MicroVM platform https://firecracker-microvm.github.io/
FreeBSD on Firecracker anouncement:

The FreeBSD Project

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org