general/other Best solution for using docker with docker compose on FreeBSD

icodeforyou said:
kpedersen it is strange though that the GitHub page of containerd itself claims that it is "available as a daemon for Linux and Windows". Maybe that is not true then.
Click to expand...
Indeed. Just clever marketing. Check out the Makefiles here:

https://github.com/containerd/containerd/blob/main/Makefile.windows#L25
https://github.com/containerd/containerd/blob/main/Makefile.freebsd#L17

Notice on those lines it is called containerd-shim? This is basically just a shim layer ultimately calling into a Linux emulator (they call this the "runtime" which is an incorrect name for it). Whereas the real meat of it is:

https://github.com/containerd/containerd/blob/main/Makefile#L84

There is a shim for Linux too because some people do opt to run the main instance in i.e Qemu.
 
So wait...

I can install nerdctl and containerd locally on FreeBSD? They are both in the ports tree I realize just now.

so getting
sysutils/nerdctl
together with
sysutils/containerd

and nerdctl would connect to a locally running containerd which is internally running a Linux emulator?

But that would mean my manual setup with the VM would become useless. Which would be great. Do I get this correctly?

Edit: Reading a bit more about it, I was way too optimistic it appears ?
 
icodeforyou said:
connect to a locally running containerd which is internally running a Linux emulator?

But that would mean my manual setup with the VM would become useless. Which would be great. Do I get this correctly?
Click to expand...
You still need to provide a runtime for containerd. The runtime is basically a Linux emulator with the correct boilerplate.
So if you create a bunch of scripts around your manual setup and give it the correct names / functions for containerd to use then you have basically created a runtime.

Another option for a runtime would be to create similar kinds of scripts around FreeBSD's jails. That would be an OCI compliant runtime but nothing from DockerHub will work because as mentioned before, they aren't fully OCI compliant.
 
I did find two more interesting things. One the one hand lima-vm, which appears to attract some people trying to get rid of docker desktop on macOS. It appears to work in principle also on FreeBSD according to this discussion and this blog post.

And then I found this PR for containerd concerning Linux containers on FreeBSD. I am not sure I understand what the functionality entails though...
 
icodeforyou said:
It appears to work in principle also on FreeBSD according to this discussion and this blog post.
Click to expand...
Keep in mind that the FreeBSD alternative to KVM is Bhyve and Qemu doesn't support Bhyve. So your Qemu VM running Linux will be very slow, falling back to the TCG JIT and not virtualization. It might be enough for you though(?).

Are there any scripts around macOS's Xhyve Docker Linux emulation? I would have thought some of them could be quite transferrable.
 
icodeforyou said:
1) Getting the docker compose command on the FreeBSD machine working properly.
Click to expand...
Slightly different direction, but devel/py-molecule has a DOCKER option nowadays. Haven't tried it yet, so I don't know how good it works. It's off by default, you will need to build it from ports to use it. You could set up some Ansible roles that will spawn any docker container you need. Then have docker itself running on a local bhyve(8) vm.
 
Just to throw this out there. I'm betting that 80% of all those wanting Docker started wanting Docker cause they read somewhere that they should want Docker but had no compelling reason or need beforehand. The same is true of Kubernetes cause Google pushed it so hard.

It's a similar effect for web technologies. Some big company announces some new tech and all of a sudden there's a whole pile of questions on Stack Overflow and other forums about how they are implementing it. Too often by sole developers who think that, if Google needs it, so do they and everyone else.
 
kpedersen You are right of course, the performance penalty on lima-vm might be too high. I might still test it though...

Any ideas on this discussion and this PR on Github? It would appear that if I understand correctly through this recent PR containerd can now detect that it is on FreeBSD and use the linux compatibility layer to run Linux container images?

drhowarddrfine the main reason I want to have docker or a working docker equivalent is for the sake of interoperability between my FreeBSD laptop and my Linux machines. I want to be able to work on the same projects and have the same containerized dev environment defined by the docker / docker-compose files on both OS.

General remark: Actually I'd say this thread has been quite a success so far. Even though the best solution has not been found yet, to there is a lot of stuff I now know I did not know before ?
 
drhowarddrfine said:
Just to throw this out there. I'm betting that 80% of all those wanting Docker started wanting Docker cause they read somewhere that they should want Docker but had no compelling reason or need beforehand. The same is true of Kubernetes cause Google pushed it so hard.

It's a similar effect for web technologies. Some big company announces some new tech and all of a sudden there's a whole pile of questions on Stack Overflow and other forums about how they are implementing it. Too often by sole developers who think that, if Google needs it, so do they and everyone else.
Click to expand...
Isn't that what's been known as a market & demand driven (self) propelling process? ;)
 
drhowarddrfine said:
We find this is the typical case of those who never used FreeBSD. Like you, those who come to learn and understand sometimes get amazed.
Click to expand...
Even though that might sound a bit odd to those who advised me in the beginning of this thread to just run my containers on Linux: What I like about working with FreeBSD is that you kinda need to look deeper at the options available when stuff doesn't just work out of the box. And in the process you learn a lot of implementation details, both about the operating system and your software, that you might never have discovered staying on Linux or even something else entirely...

And for some reason I like debugging FreeBSD more than I like debugging Linux.
 
icodeforyou said:
forquare thanks very much for these resources. nerdctl looks like it could be the way to go here. I do not particularly mind getting rid of Docker's daemon, as long as I can build a list of containers from custom dockerfiles specified in a docker-compose.yml ... I might give it a try. The only thing I cannot spot immediately is how to connect nerdctl to a remotely running instance of containerd. For docker you just have to specify an environment variable. Any ideas on that?
Click to expand...
Alas now i look into it more, it does not ? See here.

It seems when i was using nerdctl on macOS with colima, the installer actually replaced my local nerdctl with a shell script:

Code: 
.colima %>cat /usr/local/bin/nerdctl.lima
#!/bin/sh
set -eu
exec lima nerdctl "$@"

But maybe that information is good enough to you with Docker? Could you perhaps install the Docker CLI in your VM and write a small shell script to ssh to it (like the FAQ link I posted above)
 
I will be gone for a few days, then I will try out what works best I suppose. We got quite a few options now.

But I for sure need to do some testing regarding the containerd pull request I mentioned earlier which still appears to me to enable Linux containers on FreeBSD via the linux compatibility layer a.k.a linuxulator. If that solution were to work, then I would just install a local containerd via linuxulator + nerdctl on bare FreeBSD.
 
I will always suggest that people have far fewer problems if they quit trying to make FreeBSD act like Linux and use the tools of FreeBSD to work like FreeBSD.
 
drhowarddrfine I guess that makes sense from a certain perspective, but it sadly does not meet the requirement of interoperability that I have. In fact it quite clearly simply ignores it.
 
zirias@ said:
Seriously, why?

Don't get me wrong, but the most important aspect in chosing an OS is the application it must support. So if docker is a prio for you, use Linux.
Click to expand...
Agreed. The ABI in the vast majority of Docker and Podman containers are built using the Linux ABI. Some have tried to use the Windows ABI but those are few and far between.

I suppose a person could for the sake of saying they've done it build FreeBSD containers but I doubt there will be much uptake in the community.

Also remember Linux Cgroups + Namespaces is not the same as FreeBSD jails. Though cgroups + namespaces is IMO inferior to jails (it is less secure), cgroups + namespaces is the industry standard. There are differences that docker/podman containers rely on.

Adding cgroups + namespaces to the linuxlator might be the path forward but the cost in terms of reduced security compared to jails should also be considered.

But native FreeBSD docker/podman images might have limited utility.
 
icodeforyou said:
That is the whole point I am trying to make about interoperability I'd say
Click to expand...
Then you need to interoperate and use Linux.

Contrary to what cy@ mentioned, a "standard" that dictates a specific OS and software implementation can never be a standard. It might be crazy popular but then again... so is Windows which Microsoft continuously self-declares a "standard".
 
kpedersen said:
Then you need to interoperate and use Linux.

Contrary to what cy@ mentioned, a "standard" that dictates a specific OS and software implementation can never be a standard. It might be crazy popular but then again... so is Windows which Microsoft continuously self-declares a "standard".
Click to expand...
The standard is whatever most of the people accept. You and I may not like it but that's life.

If you care to notice who M$ is hiring, it's Linux developers. Windows isn't the thing it used to be. Their focus is cloud. Cloud for the reason of continuous revenue stream. (Follow the money.)

You and I will likely be surprised (or horrified) by what this industry looks like in five years.
 
I was away for a few days... no one had a useful comment on the containerd PR I already mentioned several times?

I might just try building the newest containerd from github with nerdctl and see if it actually runs linux containers now...

btw. interoperability has the prefix inter- within it which means: between different things... FreeBSD cannot inter-operate with itself, neither can Linux.

That being said: I already mentioned that I mainly(!) operate on Linux. I just also started liking FreeBSD in the not so long ago. And this thread is about finding a solution for my FreeBSD laptop to inter-operate effectively with by development set which includes docker.
 
Okay, so first feedback for anyone interested:

1) The current state of nerdctl on FreeBSD itself makes it not very useful. Neither does nerdctl compose work, nor can it handle networking between containers it appears, due to missing CNI plugin.

2) Lima-vm on the other hand actually does appear to work in principle. Building containers however is horrendously slow. Up to the point where building the app is actually not working because one container won't build. Meanwhile the whole system becomes unresponsive with the X server not moving the mouse any more. Not sure why that is.

I will later explore the other options.
 
After 2 days of hard work :D


sh: 
atag@atag-FreeBSD ~> docker info
Client:
 Version:    28.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /home/atag/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.38.1
    Path:     /home/atag/.docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 2
 Server Version: 28.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.6-0-ge89a299
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.12.36-boot2docker
 Operating System: Boot2Docker 28.3.1 (TCL 16.0)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 972.4MiB
 Name: bhyve
 ID: df419523-e30a-466d-8551-474aeb26db1b
 Docker Root Dir: /mnt/vda1/var/lib/docker
 Debug Mode: false
 Labels:
  provider=bhyve
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

atag@atag-FreeBSD ~> docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
bhyve     *        bhyve        Running   tcp://192.168.99.102:2376           v28.3.1
default   -        virtualbox   Stopped                                       Unknown
Updated boot2docker to TinycoreLinux 16.0 (Linux kernel 6.12.36), still in progress...
The docker-buildx plugin is the brebuilt freebsd binary from github (this one was the key of the upgrade)
The docker-compose plugin is the linux binary, it just works fine with the native application :-/
The docker-machine port got a tiny patch (changed the client API version string)
... still making fixes in the bhyve driver 🥹

It is going to be public soon 😉
 
Good news, since f7ee9c8 we can compile native compose module on FreeBSD 🥹


sh: 
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 28.3.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.6-0-ge89a299
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.37-boot2docker
 Operating System: Boot2Docker 28.3.2 (TCL 16.1)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 972.1MiB
 Name: v2
 ID: 8ee777a5-9a30-4091-bfb4-441b4d965ac2
 Docker Root Dir: /mnt/vda1/var/lib/docker
 Debug Mode: false
 Username: atagtm
 Labels:
  provider=bhyve
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
cgroup v2 is finally working

sh: 
docker@v2:~$ ls /sys/fs/cgroup/docker
cgroup.controllers               io.max
cgroup.events                    io.prio.class
cgroup.freeze                    io.stat
cgroup.kill                      io.weight
cgroup.max.depth                 memory.current
cgroup.max.descendants           memory.events
cgroup.procs                     memory.events.local
cgroup.stat                      memory.high
cgroup.subtree_control           memory.low
cgroup.threads                   memory.max
cgroup.type                      memory.min
cpu.idle                         memory.numa_stat
cpu.max                          memory.oom.group
cpu.max.burst                    memory.peak
cpu.stat                         memory.reclaim
cpu.stat.local                   memory.stat
cpu.weight                       memory.swap.current
cpu.weight.nice                  memory.swap.events
cpuset.cpus                      memory.swap.high
cpuset.cpus.effective            memory.swap.max
cpuset.cpus.exclusive            memory.swap.peak
cpuset.cpus.exclusive.effective  pids.current
cpuset.cpus.partition            pids.events
cpuset.mems                      pids.events.local
cpuset.mems.effective            pids.max
io.latency                       pids.peak
 
You must log in or register to reply here.
Back
Top