How to create a ZFS dataset within a jail?

I

ikevin8me

I'm trying to run Docker (must be within a FreeBSD jail)

How do I create this:
Code: 
--
You will need to create a ZFS dataset on /usr/docker
# zfs create -o mountpoint=/usr/docker <zroot>/docker
--

when zfs list shows "no datasets available" ?

The jail name is "testing" and it seems that is trying to detect "zroot/jails/testing".

Code: 
--
# docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem zroot/jails/testing: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist
--

Please help. Thanks!
 
Last edited by a moderator:
Do you have mount.devfs; in /etc/jail.conf? I used to have a ZFS jail and I believe that was necessary for ZFS to function in the jail. What are the values of these sysctl variables?
Code: 
security.jail.enforce_statfs
security.jail.mount_allowed
security.jail.mount_devfs_allowed
security.jail.mount_zfs_allowed
 
Last edited by a moderator:
I'm still using the old style rc.conf, I can't find the equivalent for:
Code: 
security.jail.enforce_statfs
security.jail.mount_zfs_allowed
It would be nice if you could tell me how to do it in rc.conf.

For these, the equivalent are:
security.jail.mount_allowed = allow.mount set from jail_<jname>_mount_enable
security.jail.mount_devfs_allowed = mount.devfs set from jail_<jname>_devfs_enable

Thanks!
 
Last edited by a moderator:
My current values are:
security.jail.enforce_statfs: 2
security.jail.mount_allowed: 0
security.jail.mount_devfs_allowed: 0
security.jail.mount_zfs_allowed: 0
 
Yeah you're going to need those sysctl values turned on. Can't do it in rc.conf. Do this in sysctl.conf:

Code: 
security.jail.mount_allowed=1
security.jail.mount_devfs_allowed=1
security.jail.mount_zfs_allowed=1

If you want to be able to ping from within your jail, turn this on as well:

Code: 
security.jail.allow_raw_sockets=1

Also add this to your jail.conf file. At the top, not in the jail definition:

Code: 
mount.devfs;

Let me know if this helps.
 
I tried to change these values in the sysctl.conf and now [the global zone] shows:
Code: 
# sysctl security.jail.mount_allowed
security.jail.mount_allowed: 1
# sysctl security.jail.mount_devfs_allowed
security.jail.mount_devfs_allowed: 1
# sysctl security.jail.mount_zfs_allowed
security.jail.mount_zfs_allowed: 1
From inside the jail, it is (I don't think this is relavant, but just in case):
Code: 
# sysctl security.jail.mount_allowed
security.jail.mount_allowed: 0
# sysctl security.jail.mount_devfs_allowed
security.jail.mount_devfs_allowed: 0
# sysctl security.jail.mount_zfs_allowed
security.jail.mount_zfs_allowed: 0

The jail always had the "jail_testing_devfs_enable="YES" in the rc.conf.

The "security.jail.enforce_statfs: 2" remained as 2. I did not change this. Do I need to change this to 0?

Alright, here is the problem: I'm trying to run Docker within a jail and I got:
"filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist" error"

It seems that I have to make 'zroot/jails/testing': dataset to exist from within the jail. How?

I have tried "zfs create -o mountpoint=/usr/docker zroot/docker" and it says "no such pool".

I also tried (from inside the jail):
# dd if=/dev/zero of=/usr/local/dockerfs bs=1024K count=4000
# zpool create -f zroot /usr/local/dockerfs
cannot create 'zroot': permission denied

Well, I'm trying to run Docker inside a jail. I believe that once resolve the ZFS dataset issue, docker will run.
 
Yes, I'm still using the old-style rc.conf and 'jail_testing_devfs_enable="YES"' had always been there.

The output from within the jail:
# mount
zroot/jails/testing on / (zfs, local, noatime, nfsv4acls)

More info (all from inside the jail):
# docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem zroot/jails/testing: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist
# zfs list
no datasets available
 
Try mounting devfs from outside the jail, then restart the jail.

mount -t devfs devfs /jails/testing/dev

Let me know if this works.
 
Yes, that worked. I mounted and I can access "/dev" from within the "testing" jail.

From inside the jail, it shows:

root@testing:/dev # ls -la
total 10
dr-xr-xr-x 9 root wheel 512 Sep 2 05:18 .
drwxr-xr-x 18 root wheel 22 Aug 24 03:13 ..
dr-xr-xr-x 2 root wheel 512 Sep 2 05:18 fd
crw-rw-rw- 1 root wheel 0x1b Sep 2 05:18 null
dr-xr-xr-x 2 root wheel 512 Sep 2 05:18 pts
crw-r--r-- 1 root wheel 0x7 Sep 2 02:15 random
lrwxr-xr-x 1 root wheel 4 Sep 2 05:18 stderr -> fd/2
lrwxr-xr-x 1 root wheel 4 Sep 2 05:18 stdin -> fd/0
lrwxr-xr-x 1 root wheel 4 Sep 2 05:18 stdout -> fd/1
lrwxr-xr-x 1 root wheel 6 Sep 2 05:18 urandom -> random
crw-rw-rw- 1 root wheel 0x1c Sep 1 21:15 zero
crw-rw-rw- 1 root operator 0x4e Sep 2 02:15 zfs
root@testing:/dev #

How can I resolve this problem (from within the jail):
# zpool create -f zroot /usr/local/dockerfs
cannot create 'zroot': permission denied
 
Sorry, I know you asked but I forgot to mention security.jail.enforce_statfs should be 0. That will allow the jail to see the mounted filesystems.

EDIT: I do not think you actually have to do what you are trying to do. Docker relies on ZFS but you only have to create that vdev backed pool if your system does not already use ZFS. Let's stick to trying to get the jail to see ZFS and not trying to create that pool.
 
This, on top of the other options we discussed, should do the trick :)
I don't remember having to do this with my last jail but I just set one up and in addition to the sysctl values and mounting devfs this must be done.

This shows the given dataset (zroot/jails/testing) to the given jail.

zfs jail [jail name] zroot/jails/testing
 
Thanks! I went past that stage. The dataset is now accessible within the jail.

However, I've got into another problem:

# service docker onestart
Starting docker...
netstat: kvm not available: /dev/mem: No such file or directory

I tried to find out but can't figure out how to mount this "/dev/mem" into the jail. Please help. Appreciated.
 
That's one of the memory files. In particular, the one that maps physical memory to file offsets. It's required for kvm to function. kvm is a library for accessing kernel data. Netstat uses kvm. This is normal for jails, as I understand it. Same thing happens to me:

[root@build11 /]# netstat
netstat: kvm not available: /dev/mem: No such file or directory


Netstat still gives me output though. Does this actually prevent docker from running?

Jails interface with the host's kernel through purpose built calls. Jails don't have their own kernel. You don't want a jail having physical memory access. The design of a jail should not allow the root of a jail to take over the physical machine.
 
Yes, it prevents Docker from running.

I execute "service docker onestart", it showed the error message and Docker stops dead. I look into "top" and did not see any Docker process running. (I am following this.)
 
Run both of these commands in your host system and in your jail. Post all four lines of output and tell me which pair came from the jail and which from the host.

uname -U
uname -K
 
From host:
# uname -U
1101001
# uname -K
1101001

From jail:
root@testing:/ # uname -U
1101001
root@testing:/ # uname -K
1101001
 
Please paste in Docker's rc.d script. That'll help me figure out how to start it without daemonizing and with a bit more verbosity. I don't think netstat is the issue. That is the normal output from within a jail. It's impossible to use KVM from within a jail. Your kernel, userland, and jail are all in sync. Something else is causing the problem.
 
You can stop docker from using netstat if you set the docker_nat_iface variable in your rc.conf. Run netstat -f inet -nrW | grep '^default' | awk '{ print $6 }' in the host environment to get the interface and set docker_nat_iface to that interface. For me it would be docker_nat_iface="em0"
 
By putting 'docker_nat_iface="em0"' into the rc.conf of the jail, I resolved the problem "netstat: kvm not available: /dev/mem: No such file or directory" (it does not show up anymore).

Docker is still failing to start because the ZFS dataset at location /usr/docker cannot be found by /proc/mount.

Well, I think I give up and would not like to try further to run Docker in a jail. It had been a great frustration. Thanks for your help very much and I'm sorry to disappoint that it does not work.
 
I have one more question:

What exactly does this 'mount -t devfs devfs /jails/testing/dev' do? It seems that it has been mounted on more than the "testing" jail (ie. affected the other jails as well).

How do I undo this mount?
 
/edit: turns out I forgot to set enforce_statfs=0. I'm beyond the below error now, but getting another error:

Code: 
root@docker:/ # docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem freenas-boot/ROOT/11.1-U5: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all freenas-boot/ROOT/11.1-U5" => cannot open 'freenas-boot/ROOT/11.1-U5': dataset does not exist.

I'll report back after investigating this further.

ComradeSlice said:
This, on top of the other options we discussed, should do the trick :)
I don't remember having to do this with my last jail but I just set one up and in addition to the sysctl values and mounting devfs this must be done.

This shows the given dataset (zroot/jails/testing) to the given jail.

zfs jail [jail name] zroot/jails/testing
Click to expand...
Sorry to revive such an old thread, but this seems pretty similar to the problems I'm fighting with at the moment.

I'm trying to get docker to run in an iocage jail. I believe I've translated the security.jail.mount_allowed=1, security.jail.mount_devfs_allowed=1, security.jail.mount_zfs_allowed=1, and mount.devfs; to their iocage setting equivalents, but now I'm stuck because I don't know how to show the dataset to the jail, or how to show the right one.

After running zfs jail 28 tank/iocage/jails/docker, docker -d still results in the following output:

Code: 
root@docker:/ # docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem tank/iocage/jails/docker/root: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all tank/iocage/jails/docker/root" => cannot open 'tank/iocage/jails/docker/root': dataset does not exist


The error actually says that the ./docker/root dataset does not exist. That's true, because at the moment that is a directory within the ./docker dataset. Does docker actually require this to be a separate dataset within ./docker? Do I have to create the ./docker/root dataset manually before creating the iocage jail? (That just seems a bit odd to me since the ./docker dataset is usually created automatically along with the jail, hence my question)
 
You must log in or register to reply here.
Back
Top