I try to run Kuberenetes in Jails but ...

logicue

logicue

Hi

I want to run Kubernetes in jails with zfs instead using a virtualisation environment.

To accomplish this I have to install a debian jessie in jail with debootstrap and install docker, etcd and so on. However, trying to install docker encounter end up with permission errors ond /dev/* The jail configurations:

/etc/rc.conf
Code: 
####### GLOBAL  #######

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
linux_enable="YES"
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
sshd_enable="YES"
powerd_enable="YES"
devfs_load_rulesets="YES"
devfs_rulesets="/etc/defaults/devfs.rules /etc/devfs.rules"


##### NETWORK #######
hostname="myHost"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"

###### VIRTUAL NETWORK ######
cloned_interfaces="vlan0"
gateway_enable="YES"
ifconfig_vlan0="inet 10.0.0.1 netmask 255.255.255.0"

#####  FIREWALL  #######
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""

###  JAILS  ######
jail_enable="YES"
jail_list="k8s-master k8s-worker-01"

/etc/jail.conf
Code: 
allow.raw_sockets = 1;
exec.start = "/etc/init.d/rc 3";
exec.stop = "/etc/init.d/rc 0";
exec.consolelog = "/var/log/jail_${name}_console.log";
devfs_ruleset=400;
mount.devfs;
mount.fstab = "/etc/fstab.$name";
mount.fdescfs;
allow.mount;
allow.set_hostname = 0;
allow.sysvipc = 1;
path = "/jailz/${name}";
interface = vlan0;

k8s-master {
        host.hostname = "k8s-master";
        ip4.addr = 10.0.0.10;
}

k8s-worker01 {
        host.hostname = "k8s-worker-01";
        ip4.addr = 10.0.0.11;
}

k8s-worker02 {
        host.hostname = "k8s-worker-02";
        ip4.addr = 10.0.0.12;
}

/etc/fstab.k8s-master
Code: 
linsys   /jailz/k8s-master/sys         linsysfs  rw          0 0
linproc  /jailz/k8s-master/proc        linprocfs rw          0 0
tmpfs    /jailz/k8s-master/run         tmpfs     rw,mode=777 0 0


So far so good. We successfully start jail:
$ sudo service jail start k8s-master
$ sudo jexec 1 /bin/bash
k8s-master# apt-get install docker-ce

And here is the problem:
Code: 
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
Setting up makedev (2.3.1-93) ...
mknod: 'mem-': Operation not permitted
makedev mem c 1 1 root kmem 0640: failed
mknod: 'kmem-': Operation not permitted
makedev kmem c 1 2 root kmem 0640: failed
mknod: 'null-': Operation not permitted
makedev null c 1 3 root root 0666: failed
mknod: 'port-': Operation not permitted
makedev port c 1 4 root kmem 0640: failed
mknod: 'zero-': Operation not permitted
makedev zero c 1 5 root root 0666: failed
ln: failed to create symbolic link 'core': Operation not permitted
mknod: 'full-': Operation not permitted
makedev full c 1 7 root root 0666: failed
mknod: 'random-': Operation not permitted
makedev random c 1 8 root root 0666: failed
mknod: 'urandom-': Operation not permitted
makedev urandom c 1 9 root root 0666: failed
mknod: 'tty-': Operation not permitted
makedev tty c 5 0 root tty 0666: failed
mknod: 'ram0-': Operation not permitted
makedev ram0 b 1 0 root disk 0660: failed
mknod: 'ram1-': Operation not permitted
makedev ram1 b 1 1 root disk 0660: failed
mknod: 'ram2-': Operation not permitted
makedev ram2 b 1 2 root disk 0660: failed
mknod: 'ram3-': Operation not permitted
makedev ram3 b 1 3 root disk 0660: failed
mknod: 'ram4-': Operation not permitted
makedev ram4 b 1 4 root disk 0660: failed
mknod: 'ram5-': Operation not permitted
makedev ram5 b 1 5 root disk 0660: failed
mknod: 'ram6-': Operation not permitted
makedev ram6 b 1 6 root disk 0660: failed
mknod: 'ram7-': Operation not permitted
makedev ram7 b 1 7 root disk 0660: failed
mknod: 'ram8-': Operation not permitted
makedev ram8 b 1 8 root disk 0660: failed
mknod: 'ram9-': Operation not permitted
makedev ram9 b 1 9 root disk 0660: failed
mknod: 'ram10-': Operation not permitted
makedev ram10 b 1 10 root disk 0660: failed
mknod: 'ram11-': Operation not permitted
makedev ram11 b 1 11 root disk 0660: failed
mknod: 'ram12-': Operation not permitted
makedev ram12 b 1 12 root disk 0660: failed
mknod: 'ram13-': Operation not permitted
makedev ram13 b 1 13 root disk 0660: failed
mknod: 'ram14-': Operation not permitted
makedev ram14 b 1 14 root disk 0660: failed
mknod: 'ram15-': Operation not permitted
makedev ram15 b 1 15 root disk 0660: failed
mknod: 'ram16-': Operation not permitted
makedev ram16 b 1 16 root disk 0660: failed
ln: failed to create symbolic link 'ram': Operation not permitted
mknod: 'loop0-': Operation not permitted
makedev loop0 b 7 0 root disk 0660: failed
mknod: 'loop1-': Operation not permitted
makedev loop1 b 7 1 root disk 0660: failed
mknod: 'loop2-': Operation not permitted
makedev loop2 b 7 2 root disk 0660: failed
mknod: 'loop3-': Operation not permitted
makedev loop3 b 7 3 root disk 0660: failed
mknod: 'loop4-': Operation not permitted
makedev loop4 b 7 4 root disk 0660: failed
mknod: 'loop5-': Operation not permitted
makedev loop5 b 7 5 root disk 0660: failed
mknod: 'loop6-': Operation not permitted
makedev loop6 b 7 6 root disk 0660: failed
mknod: 'loop7-': Operation not permitted
makedev loop7 b 7 7 root disk 0660: failed
mknod: 'tty0-': Operation not permitted
makedev tty0 c 4 0 root tty 0600: failed
mknod: 'console-': Operation not permitted
makedev console c 5 1 root tty 0600: failed
rm: cannot remove 'fd': Is a directory
ln: failed to create symbolic link 'stdin': Operation not permitted
ln: failed to create symbolic link 'stdout': Operation not permitted
ln: failed to create symbolic link 'stderr': Operation not permitted
/sbin/MAKEDEV: don't know how to make device "tty0"
dpkg: error processing package makedev (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mountall:
 mountall depends on makedev; however:
  Package makedev is not configured yet.

dpkg: error processing package mountall (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cgroupfs-mount:
 cgroupfs-mount depends on mountall (>> 2.48~); however:
  Package mountall is not configured yet.

dpkg: error processing package cgroupfs-mount (--configure):
 dependency problems - leaving unconfigured
Setting up fuse (2.9.3-15+deb8u2) ...
libkmod: ERROR ../libkmod/libkmod-module.c:1638 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
Creating fuse device...
mknod: 'fuse-': Operation not permitted
makedev fuse c 10 229 root root 0660: failed
chmod: cannot access '/dev/fuse': No such file or directory
dpkg: error processing package fuse (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 makedev
 mountall
 cgroupfs-mount
 fuse


I don't have any experience in writing devfs rules. Any idea of custom devfs rule for this ?

Thank you
 
You must log in or register to reply here.
Back
Top