jails podman within JAIL(nested containers)

P

petru garstea

Greetings,

I have been trying to use podman in a long living jail container on FreeBSD 13.2 release.
The long living container is configured with linux emulation


Code: 
zroot/bastille/jails/podman/root on / (zfs, local, noatime, nfsv4acls)
zroot/bastille/jails/podman/root/containers on /var/db/containers (zfs, local, noatime, nfsv4acls)
devfs on /compat/linux/dev (devfs)
tmpfs on /compat/linux/dev/shm (tmpfs, local)
fdescfs on /compat/linux/dev/fd (fdescfs)
linprocfs on /compat/linux/proc (linprocfs, local)
linsysfs on /compat/linux/sys (linsysfs, local)
/tmp on /compat/linux/tmp (nullfs, local, noatime, nosuid, nfsv4acls)
/usr/home on /compat/linux/home (nullfs, local, noatime, nfsv4acls)
/usr/local/bastille/releases/13.2-RELEASE on /.bastille (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /dev (devfs)
fdescfs on /dev/fd (fdescfs)

ATM, I am trying to figure what configuration is missing on the system that causes:

Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout: stderr: operation not permitted exit status 1
Click to expand...

The full log

Code: 
root@podman:~ # podman --log-level debug run --rm --os=linux docker://docker.io/alpine cat /etc/os-release
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm --os=linux docker://docker.io/alpine cat /etc/os-release)
DEBU[0000] Using conmon: "/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/db/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver zfs
DEBU[0000] Using graph root /var/db/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/db/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/db/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "zfs"
DEBU[0000] ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be START /sbin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/bastille/jails/podman/root/containers  storage-driver=zfs
DEBU[0000] ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be FINISH  storage-driver=zfs
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Using OCI runtime "/usr/local/bin/ocijail"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image docker://docker.io/alpine (policy: missing)
DEBU[0000] Looking up image "docker.io/library/alpine:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] reference "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] reference "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] Enforcing pull policy to "newer" to pull custom platform (arch: "", os: "linux", variant: "") - local image may mistakenly specify wrong platform
DEBU[0000] Loading registries configuration "/usr/local/etc/containers/registries.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for docker.io/library/alpine:latest
DEBU[0000] parsed reference into "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest"
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
DEBU[0000] Using registries.d directory /usr/local/etc/containers/registries.d
DEBU[0000] Trying to access "docker.io/library/alpine:latest"
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.docker/config.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.dockercfg
DEBU[0000] No credentials for docker.io/library/alpine found
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /usr/local/etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] Source is a manifest list; copying (only) instance sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 for current system
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
DEBU[0000] Downloading /v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
Getting image source signatures
DEBU[0000] Reading /var/lib/containers/sigstore/library/alpine@sha256=6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0/signature-1
DEBU[0000] Not looking for sigstore attachments: disabled by configuration
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Failed to retrieve partial blob: format not supported on this system
DEBU[0000] Downloading /v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
Copying blob 4abcf2066143 [--------------------------------------] 0.0b / 3.3MiB (skipped: 0.0b = 0.00%)
Copying blob 4abcf2066143 [--------------------------------------] 0.0b / 3.3MiB | 0.0 b/s
Copying blob 4abcf2066143 done   |
Copying blob 4abcf2066143 done   |
DEBU[0001] ID:62d93b96-1b16-4703-8999-a2ba584f1bc5 FINISH  storage-driver=zfs
DEBU[0001] ID:1871d56d-a96a-4a0d-8355-6688f206d776 START /sbin/zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820  storage-driver=zfs
Copying blob 4abcf2066143 done   |
DEBU[0001] mount("zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820", "/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820", "")  storage-driver=zfs
DEBU[0001] Start untar layer
ERRO[0001] While applying layer: ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0001] unmount("/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820")  storage-driver=zfs
DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94 START /sbin/zfs destroy -r zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820  storage-driver=zfs
DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94 FINISH  storage-driver=zfs
DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout:  stderr: operation not permitted exit status 1
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0001] Shutting down engines

Thanks in advance,
Petru
 
I am trying the same, running podman in a jail made with iocage, and encounter the same error. Have you found a solution since you posted?
 
i tried running podman in an Ubuntu jail
and never got it to work either

from memory i think it may have been a systemd issue

There is a Freebsd podman package

[!WARNING] The FreeBSD port of the Podman container engine is experimental and should be used for evaluation and testing purposes only.
Click to expand...

Podman Installation | Podman

Looking for a GUI? You can find Podman Desktop here.
podman.io podman.io

but i havent tried using it
id be interested in using podman on Freebsd to run invidious

Invidious

Invidious is an open source alternative front-end to YouTube.
invidious.io invidious.io

Installation - Invidious Documentation

The official Invidious documentation
docs.invidious.io docs.invidious.io
 
I'm seeing the same here. Jail for podman, permissions given:

Code: 
  children.max = 20;
  allow.mount;
  allow.mount.devfs;
  allow.mount.procfs;
  allow.mount.linprocfs;
  allow.mount.linsysfs;
  allow.mount.zfs;
  allow.mount.nullfs;
  allow.mount.tmpfs;
  allow.mount.fdescfs;
  allow.raw_sockets;
  allow.socket_af;
  allow.sysvipc;
  allow.chflags;
  enforce_statfs=1;
  devfs_ruleset=4;

  exec.created+="zfs jail web-podman zjails/podman";
  exec.release+="zfs unjail web-podman zjails/podman";

Trying to deploy a basic image to test with:

Code: 
root@web-podman:~ # podman --log-level trace run --rm --platform linux/x86_64 docker.io/debian:latest NAME=FreeBSD cat /etc/debian-version

Errors out (snippet, full log would be huge and of no relevance):

Code: 
DEBU[0001] Downloading /v2/library/debian/blobs/sha256:7d98d813d54f6207a57721008a4081378343ad8f1b2db66c121406019171805b
DEBU[0001] GET [URL]https://registry-1.docker.io/v2/library/debian/blobs/sha256:7d98d813d54f6207a57721008a4081378343ad8f1b2db66c121406019171805b[/URL]
Copying blob 7d98d813d54f [--------------------------------------] 0.0b / 47.3MiB (skipped: 0.0b = 0.00%)
Copying blob 7d98d813d54f [--------------------------------------] 0.0b / 47.3MiB | 0.0 b/s
Copying blob 7d98d813d54f done   |
DEBU[0002] ID:0109e853-5493-4579-9e53-73918cb992e8 START /sbin/zfs create -o mountpoint=legacy zjails/podman/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1  storage-driver=Copying blob 7d98d813d54f done   |
DEBU[0002] ID:0109e853-5493-4579-9e53-73918cb992e8 FINISH  storage-driver=zfs
DEBU[0002] ID:be6f9630-ec7e-4583-966a-4b03aad23caf START /sbin/zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zjails/podman/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1  storage-driver=zfs
DEBU[0002] ID:be6f9630-ec7e-4583-966a-4b03aad23caf FINISH  storage-driver=zfs
DEBU[0002] mount("zjails/podman/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1", "/var/db/containers/storage/zfs/graph/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1", "")  storage-driver=zfs
DEBU[0002] Start untar layer
ERRO[0002] While applying layer: ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0002] unmount("/var/db/containers/storage/zfs/graph/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1")  storage-driver=zfs
DEBU[0002] ID:d1148784-cf32-424b-8f16-3c3b01d9d507 START /sbin/zfs destroy -r zjails/podman/ef5f5ddeb0a6492f959cfdcfc6b0a3518e0a120db92e53ccb8225ee481e7a4a1  storage-driver=zfs
Copying blob 7d98d813d54f done   |
DEBU[0002] Error pulling candidate docker.io/library/debian:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:7d98d813d54f6207a57721008a4081378343ad8f1b2db66c121406019171805b": ApplyLayer stdout:  stderr: operation not permitted exit status 1
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:7d98d813d54f6207a57721008a4081378343ad8f1b2db66c121406019171805b": ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0002] Shutting down engines

Which makes me wonder what would prevent "untar layer" from succeeding. Also not very useful that a "trace" loglevel really doesn't show an actual trace, the proper low level actions are not shown.

Added an issue on the github page too: https://github.com/oci-playground/freebsd-podman-testing/issues/18
 
So after some digging it turns out that it's caused by the podman storage layer using "legacy" mounts, which fails in a jail:

Code: 
root@web-podman:~ # zfs create -o mountpoint=legacy zjails/podman/testtest
root@web-podman:~ # mkdir /tmp/testtest
root@web-podman:~ # mount zjails/podman/testtest /tmp/testtest
mount: zjails/podman/testtest: Operation not permitted

Anyone have an idea on how to allow a jail to mount ZFS volumes the old way?
 
hi mate

looking the the default devfs.rule for jails

Code: 
/etc/defaults/devfs.rules

Code: 
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide

it unhides both zfs and fuse

just wondering if thats any help
 
LennertVA said:
So after some digging it turns out that it's caused by the podman storage layer using "legacy" mounts, which fails in a jail:

Code: 
root@web-podman:~ # zfs create -o mountpoint=legacy zjails/podman/testtest
root@web-podman:~ # mkdir /tmp/testtest
root@web-podman:~ # mount zjails/podman/testtest /tmp/testtest
mount: zjails/podman/testtest: Operation not permitted

Anyone have an idea on how to allow a jail to mount ZFS volumes the old way?
Click to expand...
Barking up the entirely wrong tree apparently. That's a normal error to get.

Added a truss trace of podman to https://github.com/oci-playground/freebsd-podman-testing/issues/18 ; something else is causing this.
 
You must log in or register to reply here.
Back
Top