docker and linux compatability issues

Andrew D

New Member


Messages: 8

Hi there,
I am running
Code:
FreeBSD 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Sun Jan 7 21:42:48 AEDT 2018
with
Code:
Id Refs Address Size Name
1 35 0xffffffff80200000 1fe5bd0 kernel
2 1 0xffffffff82419000 2018ed zfs.ko
7 3 0xffffffff82642000 7b0f linux_common.ko
9 1 0xffffffff8264d000 42864 linux.ko
10 1 0xffffffff82690000 3c93f linux64.ko
Got a bit of an issue with docker and I was hoping someone might be able to help.
I have all the Linux compatibility kernel modules and stuff installed.
I get the following when I try and do anything with Linux images. BSD images work fine.
Code:
root@vps2:~ # docker run alpine ls
jail: exec ls: Exec format error
jail: ls: failed
or
Code:
docker run oznu/unms ls
jail: exec /init: Exec format error
jail: /init ls: failed
Does anyone have any ideas on what may be going wrong?

Thanks in advance for your time.

Andrew
 
Last edited by a moderator:

drhowarddrfine

Son of Beastie

Reaction score: 1,199
Messages: 3,184

I don't participate here anymore but why the hell would anyone running FreeBSD remotely fathom, even once, using Linux software and Docker when there are infinitely far better FreeBSD alternatives?
 
OP
OP
A

Andrew D

New Member


Messages: 8

Hi, I am quite aware of the alternative jail implementations, however my Employer wants UMNS, which will only run in a docker container from what I've read.
If I can't get it working I'll have to move this host to linux which I really don't wish to do.
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

I don't participate here anymore but why the hell would anyone running FreeBSD remotely fathom, even once, using Linux software and Docker when there are infinitely far better FreeBSD alternatives?
It's easy to use (low barrier to entry) and it allows you to work on projects (in either a Windows, FreeBSD, macOS, or Linux environment) that will be deployed on (FreeBSD, Linux, and/or Windows) using the production environment.
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

Hi there,
I am running
FreeBSD 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Sun Jan 7 21:42:48 AEDT 2018
with
Id Refs Address Size Name
1 35 0xffffffff80200000 1fe5bd0 kernel
2 1 0xffffffff82419000 2018ed zfs.ko
7 3 0xffffffff82642000 7b0f linux_common.ko
9 1 0xffffffff8264d000 42864 linux.ko
10 1 0xffffffff82690000 3c93f linux64.ko

Got a bit of an issue with docker and I was hoping someone might be able to help.
I have all the linux compatability kernel modules and stuff installed.
I get the following when I try and do anything with linux images. BSD images work fine.

root@vps2:~ # docker run alpine ls
jail: exec ls: Exec format error
jail: ls: failed
or
docker run oznu/unms ls
jail: exec /init: Exec format error
jail: /init ls: failed

Does anyone have any ideas on what may be going wrong?

Thanks in advance for your time.

Andrew
Did you follow the instructions here ( https://wiki.freebsd.org/Docker )?
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

I followed the steps and I got an Ubuntu docker image working (quite honestly I was pleasantly surprised since I did not know Docker worked on FreeBSD; I encountered issues pulling CentOS though). I also encountered issues running Alpine Linux.

Code:
/usr/home/ps ruby@2.5.0 $ docker run -i -t alpine /bin/ls
ELF binary type "0" not known.
ELF binary type "0" not known.
jail: exec /bin/ls: Exec format error
jail: /bin/ls: failed
Maybe it has something to do with the fact that Alpine Linux uses musl instead of glibc.
 
OP
OP
A

Andrew D

New Member


Messages: 8

ubuntu does indeed run. interesting. But none of the UNMS containers work, so I guess none of them run on glibc.
linux it is :( Thanks for your help.
 

herrbischoff

Active Member

Reaction score: 73
Messages: 177

The only reliable way to run Docker on FreeBSD is to virtualize Linux in bhyve and run it there. Docker has and always will be Linux-only because the underlying technology is Linux-only. You can however run the client binary (and only the client binary) as a native FreeBSD application, no Linux emulation layer required. This way, you can control the virtualized Docker instance directly from FreeBSD. Why anyone would do that instead of just using Linux to begin (if Docker is a hard requirement) with is beyond me.

BTW: Docker as a product will be gone in all but 5 years, being obviated by Kubernetes and rkt.
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

The only reliable way to run Docker on FreeBSD is to virtualize Linux in bhyve and run it there. Docker has and always will be Linux-only because the underlying technology is Linux-only. You can however run the client binary (and only the client binary) as a native FreeBSD application, no Linux emulation layer required. This way, you can control the virtualized Docker instance directly from FreeBSD. Why anyone would do that instead of just using Linux to begin (if Docker is a hard requirement) with is beyond me.
Have you heard of SmartOS (https://www.joyent.com/smartos)? There are also Windows server containers (https://blog.docker.com/2016/09/build-your-first-docker-windows-server-container/) and there's even a FreeBSD docker image. So the technology isn't Linux only IMO.

The thing is that you can run any Linux Docker image (or even an application consisting of multiple Linux Docker containers) quite easily without minimal Linux knowledge even on Windows, so why would someone not want to do the same on FreeBSD?
 

herrbischoff

Active Member

Reaction score: 73
Messages: 177

Have you heard of SmartOS
It's a question of priorities. That's a commercial platform. The vendor primarily wants to sell their product to enterprises and can therefore invest heavily into customization.

even on Windows
This is because Microsoft has put serious development manpower behind this because they realized Windows as a development platform was losing to Linux and (to a smaller but still significant degree) macOS. FreeBSD development neither has this incentive or interest nor the time or resources to spare to port some functionality that would not be maintained upstream anyway. What you're asking for is a serious commitment with very little payoff. Besides, the Docker container itself can only ever run Linux (except on Windows where it can run Windows as well). What motivation should there be for the FreeBSD community to invest significant time and effort into a Linux-based container format when jails are a comparable native and fully-tested mechanism?

You may be interested in a FreeBSD-native implementation of the App Container Specification: https://github.com/3ofcoins/jetpack (Prototype)

It hasn't been in active development for over a year now.
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

This is because Microsoft has put serious development manpower behind this because they realized Windows as a development platform was losing to Linux and (to a smaller but still significant degree) macOS. FreeBSD development neither has this incentive or interest nor the time or resources to spare to port some functionality that would not be maintained upstream anyway. What you're asking for is a serious commitment with very little payoff. Besides, the Docker container itself can only ever run Linux (except on Windows where it can run Windows as well). What motivation should there be for the FreeBSD community to invest significant time and effort into a Linux-based container format when jails are a comparable native and fully-tested mechanism?
I am not asking for anything for the record and I would not want anyone to waste their time on a futile effort. Docker on FreeBSD exists because someone wanted it and is usable even if it is experimental. Doesn't Docker on FreeBSD use jails anyway, so are you against the Linux ABI emulation in FreeBSD?

In the sentence where I said "even on Windows", I wanted to emphasize how it makes it possible for end users to use pre-configured software without having to install anything directly on the host system and without having to acquire the technical knowledge of how to configure the software themselves. See (https://github.com/jupyter/docker-stacks) and (https://github.com/sagemath/docker-images) for examples of what I mean.
 

herrbischoff

Active Member

Reaction score: 73
Messages: 177

usable even if it is experimental
For all intents and purposes it's not working. When using something like containers, you want production-level stable, not experimental. Your definition of "working" may be very different from mine though.

without having to acquire the technical knowledge
Yes, it definitely is.

so are you against the Linux ABI emulation in FreeBSD?
I'm indifferent. It's just not meant for something as deeply reliant on the Linux kernel as Docker is. If you want Docker, use Linux. If you want jails, use FreeBSD. If you're serious about containers, use CoreOS or SmartOS or any of the other (commercial) strategic platforms trying to corner the market. Your needs define your tech stack, not the other way around.
 

Preetpal

Active Member

Reaction score: 27
Messages: 117

For all intents and purposes it's not working. When using something like containers, you want production-level stable, not experimental. Your definition of "working" may be very different from mine though.
Everything is experimental for a period of time before becoming "production-level stable" (give the developers some slack; it is labelled experimental for a reason). Maybe it's a stretch to say it's usable (I have only installed Ubuntu and ran bash myself), but the potential is there. Here is a session from inside a running Ubuntu Docker container on FreeBSD.

Code:
root@freebsd-gateway:~ # uname -a
FreeBSD freebsd-gateway 11.1-RELEASE-p4 FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
root@freebsd-gateway:~ # docker run -it ubuntu /bin/bash
root@:/# cd ~
root@:/root# uptime
 11:17:17 up  4:56,  0 users,  load average: 0.38, 0.50, 0.55
root@:/root# perl -e 'print "Hello, World!\n";'
Hello, World!
root@:/root# echo 'Perl works.'
Perl works.
root@:/root# ls /
.dockerenv   .dockerinit  bin/         boot/        dev/         etc/         home/        lib/         lib64/       media/       mnt/         opt/         proc/        root/        run/         sbin/        srv/         sys/         tmp/         usr/         var/         
root@:/root# ls /b
bin/  boot/
root@:/root# ls /bin/
bash   chown  dd     dnsdomainname  false    gunzip    journalctl  loginctl  mknod   mountpoint     pidof  readlink   sed         stty       systemd               systemd-machine-id-setup        tailf     true        vdir          zcat    zfgrep  zmore
cat    cp     df     domainname     fgrep    gzexe     kill        ls        mktemp  mv             ps     rm         sh          su         systemd-ask-password  systemd-notify                  tar       umount      wdctl         zcmp    zforce  znew
chgrp  dash   dir    echo           findmnt  gzip      ln          lsblk     more    networkctl     pwd    rmdir      sh.distrib  sync       systemd-escape        systemd-tmpfiles                tempfile  uname       which         zdiff   zgrep
chmod  date   dmesg  egrep          grep     hostname  login       mkdir     mount   nisdomainname  rbash  run-parts  sleep       systemctl  systemd-inhibit       systemd-tty-ask-password-agent  touch     uncompress  ypdomainname  zegrep  zless
root@:/root# ls /usr/bin/
[           base64     chsh                  debconf-communicate     dpkg-divert              faillog    gpg-zip      install   link       lslogins          nl       perl5.22.1  pwdx          sdiff             sha256sum  sum                   tac          tput                 uptime    whoami
addpart     basename   cksum                 debconf-copydb          dpkg-maintscript-helper  fallocate  gpgsplit     ionice    linux32    lspgpot           nohup    pg          realpath      select-editor     sha384sum  systemd-analyze       tail         tr                   users     x86_64
apt         bashbug    clear                 debconf-escape          dpkg-query               find       gpgv         ipcmk     linux64    mawk              nproc    pgrep       rename.ul     sensible-browser  sha512sum  systemd-cat           taskset      truncate             utmpdump  xargs
apt-cache   bootctl    clear_console         debconf-set-selections  dpkg-split               flock      groups       ipcrm     locale     mcookie           nsenter  pinky       renice        sensible-editor   shred      systemd-cgls          tee          tset                 vmstat    yes
apt-cdrom   busctl     cmp                   debconf-show            dpkg-statoverride        fmt        head         ipcs      localectl  md5sum            numfmt   pkill       reset         sensible-pager    shuf       systemd-cgtop         test         tsort                w         zdump
apt-config  captoinfo  comm                  delpart                 dpkg-trigger             fold       hostid       ischroot  localedef  md5sum.textutils  od       pldd        resizepart    seq               skill      systemd-delta         tic          tty                  w.procps
apt-get     catchsegv  csplit                diff                    du                       free       hostnamectl  join      logger     mesg              pager    pmap        rev           setarch           slabtop    systemd-detect-virt   timedatectl  tzselect             wall
apt-key     chage      cut                   diff3                   env                      getconf    i386         last      logname    mkfifo            partx    pr          rgrep         setsid            snice      systemd-path          timeout      unexpand             watch
apt-mark    chattr     deb-systemd-helper    dircolors               expand                   getent     iconv        lastb     lsattr     namei             passwd   printenv    runcon        setterm           sort       systemd-resolve       tload        uniq                 wc
arch        chcon      deb-systemd-invoke    dirname                 expiry                   getopt     id           lastlog   lscpu      nawk              paste    printf      savelog       sg                split      systemd-run           toe          unlink               whereis
awk         chfn       debconf               dpkg                    expr                     gpasswd    infocmp      ldd       lsipc      newgrp            pathchk  prlimit     script        sha1sum           stat       systemd-stdio-bridge  top          unshare              which
base32      chrt       debconf-apt-progress  dpkg-deb                factor                   gpg        infotocap    line      lslocks    nice              perl     ptx         scriptreplay  sha224sum         stdbuf     tabs                  touch        update-alternatives  who
root@:/root# cat .
./        ../       .bashrc   .profile  
root@:/root# cat /proc/cpuinfo | head -n 7 | grep Intel
vendor_id   : GenuineIntel
model name   : Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
root@:/root# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
root@:/root# echo "Did you even try Docker on FreeBSD???"
Did you even try Docker on FreeBSD???
root@:/root# exit
exit
root@freebsd-gateway:~ # docker version
Client version: 1.7.0-dev
Client API version: 1.19
Go version (client): go1.9.2
Git commit (client): 582db78
OS/Arch (client): freebsd/amd64
Server version: 1.7.0-dev
Server API version: 1.19
Go version (server): go1.9.2
Git commit (server): 582db78
OS/Arch (server): freebsd/amd64
root@freebsd-gateway:~ #
 

herrbischoff

Active Member

Reaction score: 73
Messages: 177

We appear to have very different views on this and you seem to be intent on misunderstanding me. Best of luck in using Docker on FreeBSD. If it works for you, that's great.
 

sko

Well-Known Member

Reaction score: 226
Messages: 433

Have you heard of SmartOS
It's a question of priorities. That's a commercial platform. The vendor primarily wants to sell their product to enterprises and can therefore invest heavily into customization.
SmartOS is released under CDDL; Triton, manta and containerpilot are realeased under Mozilla PL.
What Joyent sells is cloud-services on their infrastructure which is built on top of smartOS/Triton/manta.

We are using smartOS on 2 hosts now (2 more to come) and I'm also experimenting with smartOS as an hypervisor for redundant/failover FreeBSD gateways. Crossbow is really amazing for virtualized networks.

We don't use docker, but for some linux-only/legacy software there are several LX-zones with Alpine linux running without any issues.

The Joyent stack might be overkill for small installations, but there are projects like e.g. Danubecloud or Project FiFo which also offer direct docker integration.
 

sko

Well-Known Member

Reaction score: 226
Messages: 433

CoreOS _heavily_ relies on and makes use of systemd and provides no secure multi-tenancy as it only leverages cgroups and namespaces and lots of wallpaper and duct tape (called e.g. docker or LXC) over all the air gaps inbetween...
Most importantly, systemd can't be even remotely considered production ready (just 3 random examples that popped up first and/or came to mind...), and secondly, cgroups and namespaces (combined with all the docker/LXC duct tape) might be a convenient toolset for development and offer some great features for this use case, but all 3 were never meant to provide secure isolation for containers; so they shouldn't be used in production if you want/need secure isolation and multi-tenancy (which IMHO you should always want in a production environment).

SmartOS uses zones, respectively LX-zones, for deployment of docker containers. So each container actually has his own full network stack and is safely contained within a zone. This provides essentially the same level of isolation as running a separate KVM VM for each container (which seems to be the default solution in the linux/docker world today) - but zones run on bare-metal without all the VM and additional kernel/OS/filesystem overhead the fully-fledged KVM VM drags along.



If all you want to do is running some docker containers for development or personal use/learning/amusement, just go for one of the ready-to-use Linux-based solutions like CoreOS and do what you actually wanted to achieve with these containers.
If you want to do your development with docker on FreeBSD and are willing to get your hands dirty on the way, you can/should continue trying to get docker containers properly working inside jails, document every issue you had to work around and try to fix or at least report them and commit your bug fixes upstream, so "jailed docker containers" might become a properly robust and production-ready solution one day.
If you are in the unfortunate position and have to run docker containers in a production- or multi-tenancy environment, you just _have_ to go the extra steps involved in using a properly secure and production-safe/-ready solution like e.g. smartOS.
 

herrbischoff

Active Member

Reaction score: 73
Messages: 177

so "jailed docker containers" might become a properly robust and production-ready solution one day.
Never going to happen. The App Container Specification is going to take hold before that — and it does not appear to have any serious traction as of late but I could be wrong about that. If all communities were willing to properly talk to each other and strategic interests of the big players aligned, this could be an ideal outcome. Which henceforth would also be known as the day hell froze over. It has been argued several times over the last couple of years that ZFS along with the isolation mechanisms of jails could form a superior foundation for container technology. ZFS has been transplanted to the Linux ecosystem already. In the end, they're just going to come up with some convoluted solution promising proper isolation at some point, calling it a revolution and finally have any possible lasting benefits sacrificed on the altar of systemd anyway. At least that's my experience with Linux in the past couple of years.
 

sko

Well-Known Member

Reaction score: 226
Messages: 433

You cannot avoid systemd in the Linux world, it's on all the distributions that you would use on servers (Debian, RHEL).
We have used devuan during the migration to FreeBSD and smartOS on all systems that were running debian. Alpine, which uses openRC, also has gained quite some traction especially as a base for containers or on small VMS/VPS. All our LX-Zones are also running Alpine.

If all communities were willing to properly talk to each other and strategic interests of the big players aligned, this could be an ideal outcome.
With docker completely breaking its API and toolchain every few versions it will never become a stable cross-platform solution, hence my view of docker as a development thingy, not a production safe solution. App Container definitely looks like a sane and stable solution, but in the linux ecosystem everyone seems to be obsessed with the "not invented here" syndrome and tries to come up with its own solution. Also it seems to be the norm nowadays to constantly re-invent everything and completely break things during the process. I think I missed the point when dumpster fires became a normal way of operations, but I always keep a bag of popcorn handy to watch the fallout when finally everything blows up over there....

Which henceforth would also be known as the day hell froze over.
There's so much oil getting poured into the fire, I think we will have a warm and cozy climate down there for a long time...
 
Top