Hello all,
I am trying to set up a simple jail following various tutorials and designs for the underlying file system. It works when I do all the steps manually but fails when I try to automate it with jail.conf. In particular it fails to mount devfs in the jail, which makes the jail pretty unusable. The manual steps are like this:
This is essentially installing a read-only base system, then mounting unionfs on top of it to allow changes specific to a jail, then finally adding a devfs for the jail to use. After executing those steps /j/tst/dev contains the proper set of devices, as declared by the ruleset 4.
Now I use this jail.conf script to start the jail:
I use different values for .myhost.com, ip and em0 - they are not important here. But please note the commented out lines. I can now start the jail and the devfs is still properly set up:
Now is the tricky part. I shutdown the jail, unmount all devices mounted manually above, and try to mount the filesystem automatically by commenting in either the lines with the exec.prestart/exec.poststop scripts or the mount.devfs/mount.fstab/devs_ruleset configurations in the jail.conf (above). For completeness the fstab looks like this:
In either case the devfs isn't being mounted properly. Command
Note the difference between the content of /j/tst/dev above and after I mounted the devfs manually. I see exactly the same result if I use the prestart/poststop scripts in the jail.conf as described above. It only works (and I discovered this by accident) when I add an additional line to the fstab file:
As you can see the mount shows that devfs is mounted twice. The second devfs mount somehow enforces the devices to be visible in the jail properly, as shown above.
This issue looked to me like a race condition so I tried to add some delays to exec.prestart but it didn't help. The devfs was still not being created properly. Anybody has any idea why this may be happening? Is having the devfs mounted twice as a workaround likely to cause any issues?
Greg
I am trying to set up a simple jail following various tutorials and designs for the underlying file system. It works when I do all the steps manually but fails when I try to automate it with jail.conf. In particular it fails to mount devfs in the jail, which makes the jail pretty unusable. The manual steps are like this:
Code:
zfs create tank1/j/_ro
bsdinstall jail /j/_ro
zfs create tank1/j/_tst
mkdir /j/tst
mount_nullfs -o ro /j/_ro /j/tst
mount_unionfs -o rw,noatime /j/_tst /j/tst
mount -t devfs -o rw,ruleset=4 devfs /j/tst/devThis is essentially installing a read-only base system, then mounting unionfs on top of it to allow changes specific to a jail, then finally adding a devfs for the jail to use. After executing those steps /j/tst/dev contains the proper set of devices, as declared by the ruleset 4.
Now I use this jail.conf script to start the jail:
Code:
# exec.prestart = "/sbin/mount -F /usr/local/etc/fstab/$name -a";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
# exec.poststop = "/sbin/umount -F /usr/local/etc/fstab/$name -a";
exec.clean;
# mount.devfs;
path = "/j/$name";
host.hostname = "$name.myhost.com";
# mount.fstab = "/usr/local/etc/fstab/$name";
tst {
ip4.addr = 192.168.0.5;
interface = em0;
# devfs_ruleset = 4;
}I use different values for .myhost.com, ip and em0 - they are not important here. But please note the commented out lines. I can now start the jail and the devfs is still properly set up:
Code:
[root@myhost ~]# service jail start tst
Starting jails: tst.
[root@myhost ~]# ls -l /j/tst/dev/
total 1
dr-xr-xr-x 2 root wheel 512 May 26 15:36 fd
crw-rw-rw- 1 root wheel 0x7 May 26 15:36 null
dr-xr-xr-x 2 root wheel 512 May 26 15:36 pts
crw-rw-rw- 1 root wheel 0xc May 26 10:25 random
lrwxr-xr-x 1 root wheel 4 May 26 15:36 stderr -> fd/2
lrwxr-xr-x 1 root wheel 4 May 26 15:36 stdin -> fd/0
lrwxr-xr-x 1 root wheel 4 May 26 15:36 stdout -> fd/1
lrwxr-xr-x 1 root wheel 6 May 26 15:36 urandom -> random
crw-rw-rw- 1 root wheel 0x8 May 26 10:25 zero
crw-rw-rw- 1 root operator 0x3a May 26 10:25 zfs
[root@myhost ~]# jexec tst sh
# ls -l /dev/
total 1
dr-xr-xr-x 2 root wheel 512 May 26 14:21 fd
(...)
crw-rw-rw- 1 root operator 0x3a May 26 10:25 zfsNow is the tricky part. I shutdown the jail, unmount all devices mounted manually above, and try to mount the filesystem automatically by commenting in either the lines with the exec.prestart/exec.poststop scripts or the mount.devfs/mount.fstab/devs_ruleset configurations in the jail.conf (above). For completeness the fstab looks like this:
Code:
[root@myhost ~]# cat /usr/local/etc/fstab/tst
/j/_ro /j/tst nullfs ro 0 0
/j/_tst /j/tst unionfs rw,noatime 0 0In either case the devfs isn't being mounted properly. Command
mount shows the mounts, but the content of /j/tst/dev is essentially empty. There are just two files probably created automatically when the jail was being started:
Code:
[root@myhost ~]# service jail start
Starting jails: tst.
[root@myhost ~]# mount
(...)
/j/_ro on /j/tst (nullfs, local, read-only)
<above>:/j/_tst on /j/tst (unionfs, local, noatime)
devfs on /j/tst/dev (devfs, local, multilabel)
[root@myhost ~]# ls -l /j/tst/dev/
total 1
lrwxr-xr-x 1 root wheel 12 May 26 15:26 log -> /var/run/log
-rw-r--r-- 1 root wheel 0 May 26 15:32 nullNote the difference between the content of /j/tst/dev above and after I mounted the devfs manually. I see exactly the same result if I use the prestart/poststop scripts in the jail.conf as described above. It only works (and I discovered this by accident) when I add an additional line to the fstab file:
Code:
[root@myhost ~]# echo "devfs /j/tst/dev devfs rw,ruleset=4 0 0" >> /usr/local/etc/fstab/tst
[root@myhost ~]# cat /usr/local/etc/fstab/tst
/j/_ro /j/tst nullfs ro 0 0
/j/_tst /j/tst unionfs rw,noatime 0 0
devfs /j/tst/dev devfs rw,ruleset=4 0 0
[root@myhost ~]# service jail start
Starting jails: tst.
[root@myhost ~]# mount
(...)
/j/_ro on /j/tst (nullfs, local, read-only)
<above>:/j/_tst on /j/tst (unionfs, local, noatime)
devfs on /j/tst/dev (devfs, local, multilabel)
devfs on /j/tst/dev (devfs, local, multilabel)
[root@myhost ~]# ls -l /j/tst/dev/
total 1
dr-xr-xr-x 2 root wheel 512 May 26 15:36 fd
(...)
crw-rw-rw- 1 root operator 0x3a May 26 10:25 zfs
[root@myhost ~]# jexec 33 sh
# ls -l /dev/
total 1
dr-xr-xr-x 2 root wheel 512 May 26 15:36 fd
(...)
crw-rw-rw- 1 root operator 0x3a May 26 10:25 zfsAs you can see the mount shows that devfs is mounted twice. The second devfs mount somehow enforces the devices to be visible in the jail properly, as shown above.
This issue looked to me like a race condition so I tried to add some delays to exec.prestart but it didn't help. The devfs was still not being created properly. Anybody has any idea why this may be happening? Is having the devfs mounted twice as a workaround likely to cause any issues?
Greg
