jails thinjails nullfs and additional mount points in jails

[toddg]

I'm following the handbook instructions for nullfs thin jails here.
I'd like to mount additional directories in my jails like so:

Jail1 :
/host/A -> /a

Jail2 :
/host/A -> /a
/host/B -> /b

But these mounts are particular to the jails, e.g. Jail2 doesn't want or need /host/B. So I don't want to define the mount points /a and /b in the base template. Esp as I will have many different types of jails, and they will each need different mount directories.

Is this possible? As I understand it, the mount points must exist inside the jail before the system will mount a directory to them. But it seems that 'mkdir -p /jails/nullfs/loadbalancer_lb_10_0_0_2/opt/config/caddy' (or 'mkdir -p /jails/containers/loadbalancer_lb_10_0_0_2/opt/config/caddy') is not sufficient or correct...

Here's an example jail conf that's failing:

/etc/jail.conf.d/loadbalancer_lb_10_0_0_2.jail

loadbalancer_lb_10_0_0_2 {

    # ----------------------------------------
    # networking
    # ----------------------------------------
    $ipaddr =  "10.0.0.2";
    $gw     =  "10.0.0.1";

    # ----------------------------------------
    # thinjail mounts
    # ----------------------------------------
    mount += "/jails/templates/RELEASE-15.0-base /jails/nullfs/loadbalancer_lb_10_0_0_2 nullfs ro 0 0";
    mount += "/jails/containers/loadbalancer_lb_10_0_0_2 /jails/nullfs/loadbalancer_lb_10_0_0_2/skeleton nullfs rw 0 0";

    # ----------------------------------------
    # jail specific mounts
    # ----------------------------------------
    # these are failing to mount. I've verified both source and target dirs exist. I have also tried creating these directories and
    # mounting them from the /jails/containers directory. Those also failed to mount.
    mount += "/opt/config/global/caddy /jails/nullfs/loadbalancer_lb_10_0_0_2/opt/config/caddy nullfs rw 0 0";
    mount += "/opt/data/global/caddy/db /jails/nullfs/loadbalancer_lb_10_0_0_2/var/db/caddy nullfs rw 0 0";
    mount += "/opt/config/global/caddy/www /jails/nullfs/loadbalancer_lb_10_0_0_2/var/www nullfs rw 0 0";
 
    # ----------------------------------------
    # jail dependencies
    # ----------------------------------------
    #depend="";
}

Errors

toddg@fbsdlocal ~/repos/biz/freebsd/cloud $ sudo service jail start
Starting jails:jail: loadbalancer_lb_10_0_0_2: mount: /jails/nullfs/loadbalancer_lb_10_0_0_2/opt: No such file or directory

System

toddg@fbsdlocal ~/repos/biz/freebsd/cloud $ tree /jails/nullfs/loadbalancer_lb_10_0_0_2/
/jails/nullfs/loadbalancer_lb_10_0_0_2/
├── opt
│   └── config
│       └── caddy
└── var
    ├── db
    │   └── caddy
    └── www

8 directories, 0 files

So the directories exist.

I've also tried the same thing but instead using the /jails/containers/loadbalancer_lb_10_0_0_2 directory instead of /jails/nullfs/loadbalancer_lb_10_0_0_2

I had everything working using fat jails, so I could always go back to that.

 

[Jim L.]

Your example doesn't stipulate the value of the jail's path parameter, which is important here. I'm pretty sure you won't be able to mount a filesystem that is outside the jail's path using a "mount" directive inside the jail.conf.

You might try creating a properly-formatted /etc/fstab.loadbalancer_lb_10_0_0_2 (on the host machine, not inside the jail) which I believe will be read by jail(8) before it jails the start-up process of the jail.

Convert all your jail-specific mounts to standard fstab(5) format in file /etc/fstab.loadbalancer_lb_10_0_0_2 and then reference that file in your jail.conf stanza for the specific jail they pertain to:

# Point to the specific fstab file for this jail
mount.fstab = "/etc/fstab.loadbalancer_lb_10_0_0_2";

 

[toddg]

Jim L.
Thx for responding. The $path param is defined in the /etc/jail.conf like this:

$j              = "/jails";
path            = "${j}/containers/${name}";
host.hostname   = "${name}";

exec.start      = "/bin/sh /etc/rc";
exec.stop       = "/bin/sh /etc/rc.shutdown";

allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset=5;
exec.timeout=90;
stop.timeout=90;
allow.mount.nullfs;

# Notes: gateway and ipaddr defined in the actual jail configuration files

$mask           =  "25";
vnet;
vnet.interface  =  "${ipaddr}b";

exec.prestart   =  "logger jail:prestart: trying to start jail ${name}...";

exec.prestart   += "/bin/sh /epair.sh ${ipaddr}";
exec.prestart   += "ifconfig ${ipaddr}a up descr vnet-${name}";
exec.prestart   += "ifconfig bridge0 addm ${ipaddr}a up";

exec.start      =  "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start      += "/sbin/ifconfig ${ipaddr}b ${ipaddr} netmask ${mask} up";
exec.start      += "/sbin/route add default ${gw}";
exec.start      += "/bin/sh /etc/rc";

exec.poststart  =  "logger jail:poststart: jail ${name} has started";

exec.prestop    =  "logger jail:prestop: shutting down jail ${name}";
exec.prestop    += "ifconfig ${ipaddr}b -vnet ${name}";

exec.poststop   =  "logger jail:poststop: jail ${name} has shut down";
exec.poststop   += "ifconfig bridge0 deletem ${ipaddr}a";
exec.poststop   += "ifconfig ${ipaddr}a destroy";

exec.consolelog="/var/log/jail-${name}.log";

# include all the jail files
.include "/etc/jail.conf.d/*.jail";

persist;

I've decided that thin jails are too complicated for me. I'm just going to use fat jails. It's not so bad b/c 1) they are zfs clones, and 2) I use a deployment tool that deploys new jails on software updates, rather than updating existing jails.