Solved Nested jail won't run automatically (but starts manually)

alfikmik

Member

Reaction score: 10
Messages: 58

system: FreeBSD 13.0-RELEASE

HOST:
Code:
cat /etc/jail.conf
# jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;

allow.noset_hostname;
allow.raw_sockets;
allow.sysvipc;
allow.dying;

path = "/vol/jls/${name}";
host.hostname = "${name}.bsd";

parent_jail {
    jid = 50;
    mount.devfs;
    devfs_ruleset=5;
    vnet;
    vnet.interface="v${jid}";
    exec.consolelog = "/var/log/jail_${name}.log";
    exec.prestart += "ifconfig epair0 create up";
    exec.prestart += "ifconfig epair0a inet 10.20.20.254 netmask 0xffffff00";
    exec.prestart += "ifconfig epair0a name j${jid}";
    exec.created   = "ifconfig epair0b name v${jid}";
    exec.poststop += "sleep 4";
    exec.poststop += "ifconfig v${jid} -vnet $name";
    exec.poststop += "ifconfig j${jid} destroy";
        
    allow.mount;
    allow.mount.nullfs;
    allow.mount.devfs;
    allow.mount.tmpfs;
    enforce_statfs=1;
    
    allow.dying;
    allow.chflags;
    allow.socket_af;
    children.max=20;
}

CHILD (rc.conf):
Code:
ifconfig_v50="inet 10.20.20.1 netmask 0xffffff00"
defaultrouter="10.20.20.254"
dumpdev="NO"
clear_tmp_enable="YES"
update_motd="no"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
sendmail_enable="NONE"
jail_enable="YES"
cron_enable="NO"
named_enable="NO"

CHILD (jail.conf):
Code:
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.noset_hostname;
allow.raw_sockets;
allow.sysvipc;
allow.dying;
path = "/vol/jls/${name}";
host.hostname = "v50-${name}.bsd";

nested_jail {
    jid = 51;
    interface = v50;
    ip4.addr = "v50|10.20.20.10/24";
    mount.devfs;
    devfs_ruleset=5;
    exec.consolelog = "/var/log/jail_${name}.log";
}

when i run from host service jail start parent_jail parent_jail starts but no nested_jail, when i log into parent_jail ( jexec 50 login -f root) and do service jail start nestad_jail is created

with some investignation i found that service -e doesn't show "jail":
Code:
# service -e
/etc/rc.d/cleanvar
/etc/rc.d/ip6addrctl
/etc/rc.d/netif
/etc/rc.d/virecover
/etc/rc.d/newsyslog
/etc/rc.d/os-release
/etc/rc.d/syslogd

it appears that some services depends on security.jail.jailed wchich in jailed enviroment is equal to 1

any ideas? what am i doing wrong?
 

Lamia

Aspiring Daemon

Reaction score: 241
Messages: 811

Your rc.conf should have the names of jails you want to start on boot, at least for the parent jail. Such might he required for the nested jail to.
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

Your rc.conf should have the names of jails you want to start on boot, at least for the parent jail. Such might he required for the nested jail to.

Nope, this is wrong way

Code:
jail_enable="YES"
jail_conf="/etc/jail.conf"
jail_parallel_start="NO"
jail_list="nestedjail"

that doesn't helped, for all configurations that i do before, empty jail_list means run all jails


as i mentioned before service -e in host shows jail as enabled service, but in nested jail doesn't, and it seems that have to be correlated with security.jail.jailed: 1

Code:
grep -C 5 "jailed" /usr/sbin/service
        exit $?
fi

if [ -n "$RESTART" ]; then
        skip="-s nostart"
        if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        skip="$skip -s nojail"
        fi
        [ -n "$local_startup" ] && find_local_scripts_new
        files=`rcorder ${skip} ${local_rc} 2>/dev/null`

--
fi

if [ -n "$ENABLED" -o -n "$RCORDER" ]; then
        # Copied from /etc/rc
        skip="-s nostart"
        if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
                skip="$skip -s nojail"
        fi
        [ -n "$local_startup" ] && find_local_scripts_new
        files=`rcorder ${skip} /etc/rc.d/* ${local_rc} 2>/dev/null`
fi

Code:
grep -C 5 "jailed" /etc/rc
# scripts to perform "boot-time configuration" including enabling and
# disabling rc.d scripts which appear later in the boot order.
trap "_rc_conf_loaded=false; load_rc_config" ALRM

skip="-s nostart"
if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        skip="$skip -s nojail"
        if [ `/sbin/sysctl -n security.jail.vnet` -ne 1 ]; then
                skip="$skip -s nojailvnet"
        fi
fi

Code:
grep -C 5 "jailed" /etc/rc.shutdown

# Determine the shutdown order of the /etc/rc.d scripts,
# and perform the operation
#
rcorder_opts="-k shutdown"
if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        rcorder_opts="$rcorder_opts -s nojail"
        if [ `/sbin/sysctl -n security.jail.vnet` -ne 1 ]; then
                rcorder_opts="$rcorder_opts -s nojailvnet"
        fi
fi
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

For now working solution is:

1. disable nested jail in rc.conf (in parent jail - jail_enable="NO")

2. in host add to /etc/jail.conf:

Code:
exec.poststart="jexec parentjail service jail onestart";
exec.prestop="jexec parentjail service jail onestop";

but I think there have to be a better way
 

Zirias

Son of Beastie

Reaction score: 1,774
Messages: 3,015

Unfortunately, it looks like this is "by design". /etc/rc.d/jail contains the following metadata block:
Code:
# PROVIDE: jail
# REQUIRE: LOGIN FILESYSTEMS
# BEFORE: securelevel
# KEYWORD: nojail shutdown

The keyword nojail means rc skips this script when running inside a jail.
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

Unfortunately, it looks like this is "by design". /etc/rc.d/jail contains the following metadata block:

So I'm curious why it is not well documented, and Hierarchical Jails section in jail(8) doesn't mention about any limitations or special way to run nested jails
 

Zirias

Son of Beastie

Reaction score: 1,774
Messages: 3,015

So i'm curious why it is not well documented, and Hierarchical Jails section in jail(8) doesn't mention about any limitations or special way to run nested jails
This manpage doesn't touch the init script at all. The nojail keyword is documented in rc(8) (but I admit, you kind of have to know that to find it...).

I guess it makes some sense as nested jails are typically started ad-hoc (one usecase is e.g. a jailed poudriere). But in your case, you could also decide to just remove this keyword. etcupdate(8) should make merging that change on updates a piece of cake.
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

this is very interesting:

https://github.com/freebsd/freebsd-src/blame/main/libexec/rc/rc.d/jail

1637588120244.png
 

Zirias

Son of Beastie

Reaction score: 1,774
Messages: 3,015

Ah, so someone changed their minds ;) Well then, just remove the keyword, and with 14, it will be gone anyways.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 13,121
Messages: 39,712

I remember Dan Langille posting something on twitter regarding jails in jails. Can't find it anymore though. He was running into the same issue, those "sub" jails don't want to start automatically.
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

ok so :) changes is more:

Diff:
diff /etc/rc.d/jail jail
9c9
< # KEYWORD: nojail shutdown
---
> # KEYWORD: shutdown
122a123
>       eval _jconfdir=\"/etc/jail.conf.d/${_j}.conf\"
127a129,131
>                       return 0
>               elif [ -r "$_jconfdir" ]; then
>                       _conf="$_jconfdir"

but in my case after remove nojail from /etc/rc.d/jail everything works like a charm
 

Alain De Vos

Son of Beastie

Reaction score: 870
Messages: 2,826

Jails within jails is speciality.
Danny Langville & Michael W Lucas could elaborate ?
 
OP
alfikmik

alfikmik

Member

Reaction score: 10
Messages: 58

Jails within jails is speciality.
Danny Langville & Michael W Lucas could elaborate ?

Yup, MWL do the job, I have all his books including "FreeBSD Mastery: Jails" where he wrote some about hierarchical jails, also freebsd docs have some info, but ... problem that is solved here isn't described anywhere.

also problem with hanging jails (often with system crash) in some cases when we use VNET (source here), is not solved and not described in books or docs i mean hack with sleep:

Code:
exec.poststop += "sleep 4";
exec.poststop += "ifconfig v${jid} -vnet $name";
exec.poststop += "ifconfig j${jid} destroy";
 
Top