hierarchical jails

[leveche]

Hello all,

I have some rather naiive questions re: hierarchical jails. For a start, when I try to start a jail within a jail I get

root@outer:/ # jail -c inner 
mount: .: Operation not permitted
jail: test1: /sbin/mount -t devfs -oruleset=4 . /jail/inner/dev: failed

My setup is as follows:

/etc/jail.conf on the host:

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
host.hostname = "$name.mylab.example.org";
path = /jail/$name;
interface = em1;
ip4.addr = 10.1.1.$ip;

outer {
    children.max = 128;
    ip4.addr     = 10.1.1.4,10.1.1.5;
    allow.mount  = true;
    allow.mount.devfs  = true;
    allow.mount.zfs  = true;
}

In the 'outer' jail, I can verify that the mount_devfs_allow parameter is set, but not sure what the security.jail.param.allow settings are:

security.jail.mount_procfs_allowed: 0
security.jail.mount_zfs_allowed: 1
security.jail.mount_devfs_allowed: 1
security.jail.param.allow.mount.procfs: 0
security.jail.param.allow.mount.zfs: 0
security.jail.param.allow.mount.devfs: 0
security.jail.param.allow.mount.: 0
security.jail.param.devfs_ruleset: 0
security.jail.devfs_ruleset: 0
security.jail.mount_allowed: 1

Is there something obvious I'm missing?

In addition, I'm confused how the hierarchical jails are meant to work with zfs jailed property: would the child jails be able to mount the child zfs datasets of the 'outer' jail? The documentation suggests otherwise. Is the hierarchical jail feature used much at all, and what are the typical scenarios? For me, it seems appealing to arrange jails into hierarchies like work.{project1,project2,...}, labs.{test1, test2,...} etc - is there anyone on this forum who is using a similar setup?

Thanks!
leveche

 

[zirias@]

/etc/jail.conf on the host:
[...]

There's probably a lot missing. For my poudriere jail, I have the following settings:

     children.max = 20;
     allow.mount;
     allow.mount.devfs;
     allow.mount.procfs;
     allow.mount.linprocfs;
     allow.mount.zfs;
     allow.mount.nullfs;
     allow.mount.tmpfs;
     allow.raw_sockets;
     allow.socket_af;
     allow.sysvipc;
     allow.chflags;
     enforce_statfs=1;

Of course, some of these options are not always needed, e.g. procfs and linprocfs. You'll need nullfs very often though, it's typically used for setting up a jail.

not sure what the security.jail.param.allow settings are

These are sysctl(8)s setting default security settings for jails. I'd suggest leaving them all 0 (default), you can override them in /etc/jail.conf for individual jails. Deny by default and allow where needed is always the best policy for anything security-related.

Is the hierarchical jail feature used much at all, and what are the typical scenarios?

I only use it to have ports-mgmt/poudriere in a jail, and this is the only usecase that actually comes to my mind. Most of the times, you just want to isolate a service, so nesting jails doesn't make much sense. But maybe you find another usecase ...

 

[Arne]

Is the hierarchical jail feature used much at all, and what are the typical scenarios? For me, it seems appealing to arrange jails into hierarchies like work.{project1,project2,...}, labs.{test1, test2,...} etc - is there anyone on this forum who is using a similar setup?

Yes, this is a very common use case. Today "normal" server hardware is extremly overpowered. We use some of our servers as "jail-farms" and bundle jails belonging to the same lab environment together under the lab master server. This helps to give stuff members the rights to restart the jails they need access to without compromising other lab environments.

VNET Jails work perfect inside hierarchical jails and helps to solve a few permisson problems.

Thinking only at poundriere is way too short. Poudriere is the special case here.