general/other Best solution for using docker with docker compose on FreeBSD

So far my little experiment of using FreeBSD for development work has been very interesting and successful in many aspects. However I might be facing a major obstacle: I am developing a TypeScript application that depends on several services (redis, postgres, potentially RabbitMQ in conjunction with other TypeScript or Python services) and I absolutely need a working reliable docker setup running on FreeBSD. Any help on getting this to work is highly appreciated!

My initial idea was to virtualize the docker daemon and then access it remotely from the FreeBSD host. This where I am standing right now:

I am running a bhyve VM on my FreeBSD laptop with Debian installed and docker enabled, running with a configuration for remote network access. The daemon is successfully accessible via network. The docker-cli on the freebsd host, which is installable via this port, can connect to the remote docker daemon and run containers (such as hello-world) on the Debian VM. I can shutdown and restart the VM and the docker client just works. Which is really nice.

However there are some problems.

I) The docker client available in ports is coming without the docker compose command. I am not quite sure why, however it might simply be out of date. There is an older version of standalone docker-compose in ports. That one however is simply too much behind to understand my current YAML files. Yet since my TypeScript app has several component services, I need a recent docker compose to manage this in development. I was thinking about building the newest version of docker cli from source but I haven't tried yet... any ideas on this issue?

II) The filesystem on the host must be accessible to Docker inside the VM in order to update on changes. So that the hot-reload functionality of ts-node running inside the application container on the Debian VM (yes it is getting a little crazy) can pick up changes in the codebase on the FreeBSD host without me having to rebuild the container. My thought was that this might require mapping the local ZFS filesystem into the Linux VM somehow. However I am not sure at this point.

Is it possible to share the filesystem between guest and host on bhyve? Or if not, how about Virtualbox instead of bhyve? I was also thinking about using either sshfs or nfs to do this. However how reliable is this going to be once we are talking persisting data between container runs?

Any suggestions helping to resolve these two main problems are highly welcome and appreciated! Also, since I consciously chose to try FreeBSD and my goal is to push it as far as possible here, I do not mind the time and effort that might be needed to get a more complicated solution working. The main thing is: Is it doable?

(Please note that this is NOT a thread on FreeBSD native alternatives to Docker. The setup must be interoperable with my existing Linux-based development workflow within the same git repo without major reconfiguration. Any FreeBSD native solution isn't really feasible here, at least to my knowledge)

Thanks in advance!
 
Seriously, why?

Don't get me wrong, but the most important aspect in chosing an OS is the application it must support. So if docker is a prio for you, use Linux.
 
Well, why not? :) People are running docker desktop on Windows and MacOS with some virtualization solution in the background if I am not mistaken. Some people are even using WSL on Windows to get a Linux environment going, without installing Linux as their primary OS.

And I can't imagine I am the only one that wants to have a working solution for docker on FreeBSD.
 
People are running docker desktop on Windows and MacOS with some virtualization solution in the background if I am not mistaken.
Run a Linux host on bhyve(8), run your kubernetes, docker or some of the other container products on that virtualized Linux host. Same solution.
 
SirDice Sure, that is what I am trying to achieve. And the basics are working already. However I need to find a solution to the two problems mentioned above. Which is why I created this thread.

These are: 1) Getting the docker compose command on the FreeBSD machine working properly. 2) Sharing the file system between FreeBSD host and Debian guest to let the containers react to updates in the codebase without rebuilding them.
 
These are: 1) Getting the docker compose command on the FreeBSD machine working properly. 2) Sharing the file system between FreeBSD host and Debian guest to let the containers react to updates in the codebase without rebuilding them.
I had similar requirements which lead me to simply using RDP.
The Linux VM runs net/xrdp and the FreeBSD host runs net/xfreerdp. Then, setup a shared folder via RDP which is very easy to accomplish by using the /drive:share parameter of xfreerdp:

Code:
xfreerdp \
    /u:jbo \
    /v:192.168.1.143:3389 \
    /dynamic-resolution \
    /clipboard \
    /drive:share,/usr/home/jbo/rdp_drive

This way, your Linux VM gets access to your hosts filesystem via RDP (in the example above to the /usr/home/jbo/rdp_drive directory). From there, either have symlinks on your hosts "shared directory" to your project directory or simply point the drive share path to the project directory itself.
Then you can also run your docker compose command in the Linux VM as it now has access to those files form the host's filesystem. Setup a filewatcher on the Linux VM to run stuff automagically if needed when files on the host OS change.
 
jbodenmann That looks interesting! I have not seen this approach before. And yes you are right, once the file system sharing is working I could run docker compose just inside the VM for the same effect.

However does my Debian guest need to run an X server for this? So far it does not.
 
However does my Debian guest need to run an X server for this? So far it does not.
I have no idea. I have this setup working successfully with a variety of Linux VMs. Most recently Ubuntu 22. No idea whether all of them happen to run X or whether there is some automatic magic involved which just makes it work despite the Linux distribution running Wayland. The reason I didn't look into this is because seemingly every time you spend time learning/understanding details of Linux or a Linux distro, you blink your eyes and everything changed already again.

All I did was installing the xrdp package and enabling the service via beloved systemd:
Code:
sudo apt install xrdp
sudo systemctl status xrdp
 
Alright, I'm gonna give it a try tomorrow and report back! ;)

Side note: My question was not so much motivated by the X vs. Wayland issue. Rather my Debian VM is as minimal as possible right now. I has literally nothing in it that is not required for docker. So no X Server as far as I know. But we shall see...
 
So no X Server as far as I know.
Well, there are plenty of options to choose from then.
For example, you can setup file synchronization between the host and the Linux VM.
Or use something like sshfs(1) to seemingly "integrate" the guets's filesystem into the host (or vice versa).
Or just setup a workflow with net/rsync.

Really, plenty of options.
 
Yeah, sshfs might have nasty lag too it before updating file system changes I learned. Same goes for file synchronization tools... not sure what you are thinking about here, but if you thinking about syncthing then I would stay away from that as well. Since the whole point of hot reload is it's hotness after all :D

I will try your solution tomorrow and see how it goes.
 
Well, why not? :) People are running docker desktop on Windows and MacOS with some virtualization solution in the background if I am not mistaken. Some people are even using WSL on Windows to get a Linux environment going, without installing Linux as their primary OS.
Docker on WSL(2) is virtualization. You are running a Linux install on Hyper-V.
Docker on macOS is virtualization. Actually using Xhyve forked from our Bhyve.
And I can't imagine I am the only one that wants to have a working solution for docker on FreeBSD.
There is no working solution for any OS other than Linux. Docker *is* Linux. None of these solutions actually provide a working solution for Docker; they simply run Linux in an emulator.

Which raises the question of why you want to run Linux in an emulator? Just run it on bare metal. More performance, less faff with passthrough.
 
One more thing to consider, is using ansible. This way if you want; you can have the configs on FreeBSD and let ansible copy the configs to the linux vm (keeping the files in sync) and run any docker commands that you need.
 
kpedersen

I might not quite get your point maybe. I primarily am running Docker just on Linux.

I just happen to have one Laptop that runs FreeBSD. And I want to develop on that Laptop exactly as much as I do on my Linux machines. Hence the need / desire to get something working.

And yeah, what I am trying on FreeBSD is essentially the DIY way of what you get nicely pre-packaged on Windows or macOS.

Edit: I also do not quite understand the bare-metal argument. Since this would clearly go against containerization all together if you stick to it until the end. Why run Redis and Postgres and Nodejs in a container at all? Why not just install Postgres? The resource efficiency is much better... And in some cases that might be a valid argument, but there are pros and cons to all these things. Having stuff reproducible in containers with declarative files and across development platforms is nice. And the performance downside of having a VM is not that bad in development either... and we are not talking about running this in production.

ct85711

I actually hadn't thought about that, even though I use ansible sometimes. I am not sure though that ansible would be nicer than docker compose during development...
 
The key part on ansible, is that it works as a generic manager delegating the various tasks out. It doesn't really take the docker compose out of the steps; it is more of that ansible delegates the docker compose steps out (so you don't have constantly make sure you have the current configs when you are testing something). This helps more when you need to communicate between multiple VM's/tasks on keeping everything organized.
 
Edit: I also do not quite understand the bare-metal argument. Since this would clearly go against containerization all together if you stick to it until the end.
The benefit of containerization is that it can be on bare metal. You also share a single kernel and with the host so you can avoid that stuff mentioned earlier such as sshfs, X11 forward, emulated hardware, etc.

Yes, some guys do run containers in a VM too but it doesn't require that. There are some benefits for and against certainly.

Obviously on FreeBSD we have Jails as our native container system. I don't know if I would specifically recommend people run them in a Bhyve (or VirtualBox) instance. It seems unnecessary for many use-cases.

Why run Redis and Postgres and Nodejs in a container at all?
For me personally, I tend to run things like Nodejs in a Jails container just to encapsulate all the millions of shite dependencies the software inevitably drags in from NPM ;)
 
kpedersen

Sure, I agree: containerization on the operating system that your container is designed for is preferable to containerization within a VM in most cases. And I would not run jails inside a bhyve VM except in exceptional circumstances. Nor do I run Docker inside a kvm on Linux.

However the whole thing is of course that I don't think jails are gonna cut it here, for the simple reason that they are not Docker. Podman on Fedora maybe would, as it is interoperable for the most part with Docker. The main point being that I am seeking a development workflow that is cross platform Linux-FreeBSD.

But I am not very experienced with jails tbh. Is there something like declarative configuration with compose and dockerfiles and a public image repository like docker hub to spin up a more complex application quickly?

ct85711

My current workflow on Linux is just: run a bunch of containers the app depends on with docker compose. The app container itself listens to changes in a folder on the local file system. Inside the app container we have ts-node running and checking if files are changed. And it will detect that because a certain folder inside the container is mapped to the source directory outside the container where I am writing my stuff.

Now on FreeBSD I need a VM in between and an additional mapping between a folder on the FreeBSD host and a folder on the VM guest. So the source directory from the FreeBSD host where I am writing the code needs to be accessible inside the Linux guest VM. And possibly without much of a delay. Because hot-reloading means that the app restarts immediately whenever you save changes to your codebase.

I don't really see where ansible would come in here.
 
However the whole thing is of course that I don't think jails are gonna cut it here, for the simple reason that they are not Docker. Podman on Fedora maybe would, as it is interoperable for the most part with Docker. The main point being that I am seeking a development workflow that is cross platform Linux-FreeBSD.
The issue is that DockerHub and alternatives are non-standard and only provide Linux binaries instead of the wide platform support required by OCI container standard.
Podman will work but *only* on Fedora because Fedora is Linux. If Podman (or Docker) was run on other operating systems, the non-standard DockerHub containers would still not work unfortunately.
But I am not very experienced with jails tbh. Is there something like declarative configuration with compose and dockerfiles and a public image repository like docker hub to spin up a more complex application quickly?
If it is the frontend scripts from Docker you like, then we do have a frontend compatible substitute available called Focker (https://github.com/sadaszewski/focker)

But again, until you can find a DockerHub-style repo with correct images, you will still have to make all your own (possibly made easier via the supported scripts if they themselves don't drag in non-standard Linux-only blobs).

You may want to contact DockerHub and ask why they are not correctly adhering to the OCI container standard. My guess is limited manpower, funds and knowledge.
 
kpedersen I just checked the image specifications defined by the OCI. As far as I can see you can just limit your platform support to Linux. There appears to be nothing that would mandate support for multiple platforms when creating container images. Also the OCI is tightly connected to the Linux Foundation. So hardly surprising that FreeBSD is not their top priority.

However I did learn that there is an implementation of containerd for Windows now.

Also bear in mind: All I really need from the FreeBSD side are two things. An up-to-date docker client (!) to connect to a remote docker daemon inside the VM and a good solution for mapping host to guest file systems into a Debian VM. I have zero problem with running Linux containers in a Linux VM on a FreeBSD host. I think for development that can be a good workable solution.

And regarding Focker, that thing has a very cringeworthy custom licencing policy based on the very peculiar political ideology of its creator. So it is basically unusable for anything where these legal questions start to matter.
 
kpedersen I just checked the image specifications defined by the OCI. As far as I can see you can just limit your platform support to Linux. There appears to be nothing that would mandate support for multiple platforms when creating container images.
Can you point that statement out in the spec? Otherwise it should probably called the LCI (Linux Container Initiative) right? ;)

From what I can see in their FAQ: https://opencontainers.org/faq/#will-the-runtime-and-image-format-specs-support-multiple-platforms

DockerHub and OCI have basically failed a requirement. Because there is exactly zero other platforms that their spec/images can support. Its a bit like Microsoft saying Microsoft Office 2016 supports *all* platforms and then only releasing Windows x86 binaries and telling people to use Windows emulators!

Also the OCI is tightly connected to the Linux Foundation. So hardly surprising that FreeBSD is not their top priority.
Some might say it isn't a standard at all and is infact a buzzword for running Linux blobs on Linux.

However I did learn that there is an implementation of containerd for Windows now.
Unfortunately there isn't. Windows runs a Linux emulator to correctly run a wrapper around containerd (a Linux program). A crap solution, just like everyone else.
https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/containerd

And regarding Focker, that thing has a very cringeworthy custom licencing policy based on the very peculiar political ideology of its creator. So it is basically unusable for anything where these legal questions start to matter.
Haha. True but you did mention this was just for personal development on your FreeBSD laptop. Any production cases will (need to be) run on Linux anyway (by design of using Linux binaries).
 
I) The docker client available in ports is coming without the docker compose command. I am not quite sure why, however it might simply be out of date. There is an older version of standalone docker-compose in ports. That one however is simply too much behind to understand my current YAML files. Yet since my TypeScript app has several component services, I need a recent docker compose to manage this in development. I was thinking about building the newest version of docker cli from source but I haven't tried yet... any ideas on this issue?
sysutils/nerdctl has been added to the Ports tree in the past few of days, which is a "Docker-compatible CLI for containerd" that supports Docker compose.

sysutils/nerdctl requires a containerd runtime, which I don't think Docker's Moby is, so you'd need to change your VM to use it.

oOiOo has also just posted the the runj project which I was about to mention also!
 
There is currently work in progress with sysutils/runj from Samuel Karp.
Just to add to this in case anyone is unsure. This will *not* allow you to run containers off DockerHub because the software in them is still compiled for the wrong operating system.

Likewise if you make Docker containers using this (once it is perhaps more mature), your images won't be able to run on Linux.
 
forquare thanks very much for these resources. nerdctl looks like it could be the way to go here. I do not particularly mind getting rid of Docker's daemon, as long as I can build a list of containers from custom dockerfiles specified in a docker-compose.yml ... I might give it a try. The only thing I cannot spot immediately is how to connect nerdctl to a remotely running instance of containerd. For docker you just have to specify an environment variable. Any ideas on that?

and also thanks to oOiOo this runj project looks very promising in the longer run for getting into a development setup based on jails. I will take a look at it.

kpedersen it is strange though that the GitHub page of containerd itself claims that it is "available as a daemon for Linux and Windows". Maybe that is not true then.
 
Back
Top