jails VNET jail with exec.jail_user

B

bbailey1024

I'm trying to set up a VNET jail that runs a single process as an unprivileged user via exec.jail_user. I'm running into difficulties getting the address and routing tables set up inside the jail. As an unprivileged user, I cannot add a ifconfig and route commands to exec.start as described in the handbook. The only option I can see is to allow the jail to be run as run (i.e., omit exec.jail_user) then drop privileges using su -m username /path/to/app or daemon -u username /path/to/app. Those options seem like hacks and not the intended way to accomplish this.

I attempted to be clever by adding jexec commands using exec.created, but this indicated that the interface does not exist in the jail, which I found confusing. Here is a config with my attempts. It's likely that I'm just doing something wrong or overlooked something in the documentation, but I'm not sure how to proceed. Any advice, troubleshooting tips, or solutions would be appreciated.

Bash: 
# freebsd-version -kru
15.0-RELEASE-p4
15.0-RELEASE-p4
15.0-RELEASE-p4

Bash: 
postgres {
  # Logging
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # Permissions
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # Postgres requires access shared memory
  # The new setting will provide the jail its own namespace
  sysvshm = new;

  # Path/Hostname
  path = "/jail/${name}";
  host.hostname = "${name}";

  # User
  exec.jail_user = "postgres";

  # Debug
  persist;

  # Mounts
  mount = "/zdata/postgres /jail/postgres/var/db/postgres nullfs rw 0 0";

  # VNET
  vnet;
  vnet.interface = "${epair}b";

  # Networks/Interfaces
  $id = "11";
  $ip = "10.10.80.${id}/24";
  $gateway = "10.10.80.1";
  $bridge = "vnet80";
  $epair = "epair${id}";

  # Host network setup
  exec.prestart += "/sbin/ifconfig ${epair} create up";
  exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";

  # Jail network setup - Added an ifconfig -a for debugging
  exec.created  = "jexec ${name} /bin/sh -c '/sbin/ifconfig -a'";
  exec.created += "jexec ${name} /bin/sh -c '/sbin/ifconfig ${epair}b ${ip} up'";
  exec.created += "jexec ${name} /bin/sh -c '/sbin/route add default ${gateway}'";

  # Jail network setup - works fine without exec.jail_user
#  exec.start    = "/sbin/ifconfig ${epair}b ${ip} up";
#  exec.start    += "/sbin/route add default ${gateway}";

  #exec.start    = "/usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile start";
  #exec.stop     = "/usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile stop";

  exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

If I attempt to start the jail using the above config, I receive the following:
Bash: 
# jail -v -c postgres
postgres: run command: /sbin/mount -t nullfs -o rw /zdata/postgres /jail/postgres/var/db/postgres
postgres: run command: /sbin/mount -t devfs -oruleset=5 . /jail/postgres/dev
postgres: run command: /sbin/ifconfig epair11 create up
epair11a
postgres: run command: /sbin/ifconfig epair11a up descr jail:postgres
postgres: run command: /sbin/ifconfig vnet80 addm epair11a up
postgres: jail_set(JAIL_CREATE) name=postgres allow.raw_sockets devfs_ruleset=5 sysvshm=new path=/jail/postgres host.hostname=postgres persist vnet=new
postgres: created
postgres: run command: /bin/sh -c jexec postgres /bin/sh -c '/sbin/ifconfig -a'
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
postgres: run command: /bin/sh -c jexec postgres /bin/sh -c '/sbin/ifconfig epair11b 10.10.80.11/24 up'
ifconfig: interface epair11b does not exist
jail: postgres: /bin/sh -c jexec postgres /bin/sh -c '/sbin/ifconfig epair11b 10.10.80.11/24 up': failed
postgres: removed
postgres: run command: /sbin/umount /jail/postgres/dev
postgres: run command: /sbin/umount -t nullfs /jail/postgres/var/db/postgres
 
I have only just tried messing about with executing programs in jails and have found that jexec does not work in an iocage jail.

Instead I need to run something like

iocage exec JAILNAME ifconfig

as an example.
 
Maybe you should set up doas or mdo (mac_do) to execute ifconfig and route as root user in the jail.

Alternatively, remove vnet.interface=epair11b and do it manually in exec.created and exec.prestop, like:
Code: 
exec.created += "/sbin/ifconfig ${epair}b vnet ${name}";
exec.created+= "jexec ${name} /sbin/ifconfig ${epair}b up";
exec.created += "jexec ${name} /sbin/ifconfig ${epair}b inet 10.10.80.${id}/24";
exec.created += "jexec ${name} /sbin/route add default ${gateway}";
exec.prestop += "jexec ${name} ${epair}b down";
exec.prestop += "/sbin/ifconfig ${epair}b -vnet ${name}";
 
Thanks Hiroo! Manually moving the interface to the jail during exec.created instead of via vnet.interface did the trick. I skipped the exec.prestop steps as I'm destroying the epair in exec.poststop and that seems to clean things up (I may add the exec.prestop cleanup later just to be explicit). I'm also using ifconfig and route from the host with the -j flag instead of jexec. Once I moved things around it worked perfectly. I really appreciate the help.

Here's the working config and some output for reference.
Bash: 
postgres {
  # Logging
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # Permissions
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # Postgres requires access shared memory
  # The new setting will provide the jail its own namespace
  sysvshm = new;

  # Path/Hostname
  path = "/jail/${name}";
  host.hostname = "${name}";

  # User
  exec.jail_user = "postgres";

  # Debug
  # persist;

  # Mounts
  mount = "/zdata/postgres /jail/postgres/var/db/postgres nullfs rw 0 0";

  # VNET
  vnet;

  # Networks/Interfaces
  $id = "11";
  $ip = "10.10.80.${id}/24";
  $gateway = "10.10.80.1";
  $bridge = "vnet80";
  $epair = "epair${id}";

  # Host network setup
  exec.prestart  = "/sbin/ifconfig ${epair} create up";
  exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";

  # Jail network setup
  exec.created  = "/sbin/ifconfig ${epair}b vnet ${name}";
  exec.created += "/sbin/ifconfig -j ${name} ${epair}b ${ip} up";
  exec.created += "/sbin/route -j ${name} add default ${gateway}";

  exec.start    = "/usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile start";
  exec.stop     = "/usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile stop";

  exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

Bash: 
# jail -v -c postgres
postgres: run command: /sbin/mount -t nullfs -o rw /zdata/postgres /jail/postgres/var/db/postgres
postgres: run command: /sbin/mount -t devfs -oruleset=5 . /jail/postgres/dev
postgres: run command: /sbin/ifconfig epair11 create up
epair11a
postgres: run command: /sbin/ifconfig epair11a up descr jail:postgres
postgres: run command: /sbin/ifconfig vnet80 addm epair11a up
postgres: jail_set(JAIL_CREATE) persist name=postgres allow.raw_sockets devfs_ruleset=5 sysvshm=new path=/jail/postgres host.hostname=postgres vnet=new
postgres: created
postgres: run command: /sbin/ifconfig epair11b vnet postgres
postgres: run command: /sbin/ifconfig -j postgres epair11b 10.10.80.11/24 up
postgres: run command: /sbin/route -j postgres add default 10.10.80.1
add net default: gateway 10.10.80.1
postgres: run command in jail as postgres: /usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile start
postgres: jail_set(JAIL_UPDATE) jid=38 nopersist

Bash: 
# jexec postgres ifconfig -a
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair11b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=60000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 58:9c:fc:10:63:24
        inet 10.10.80.11 netmask 0xffffff00 broadcast 10.10.80.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

# jexec postgres netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            10.10.80.1         UGS        epair11b
10.10.80.0/24      link#5             U          epair11b
10.10.80.11        link#6             UHS             lo0

# jexec postgres ping -c 4 10.10.80.1
PING 10.10.80.1 (10.10.80.1): 56 data bytes
64 bytes from 10.10.80.1: icmp_seq=0 ttl=64 time=0.033 ms
64 bytes from 10.10.80.1: icmp_seq=1 ttl=64 time=0.036 ms
64 bytes from 10.10.80.1: icmp_seq=2 ttl=64 time=0.031 ms
64 bytes from 10.10.80.1: icmp_seq=3 ttl=64 time=0.058 ms

--- 10.10.80.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.031/0.039/0.058/0.011 ms

Bash: 
# ps -J postgres -u
USER       PID %CPU %MEM    VSZ   RSS TT  STAT STARTED    TIME COMMAND
postgres 11133  0.0  1.7 195224 35596  -  IsJ  16:09   0:00.02 /usr/local/bin/postgres -D /var/db/postgres/data
postgres 28515  0.0  1.1  41828 22120  -  IsJ  16:09   0:00.00 postgres: logger  (postgres)
postgres 29182  0.0  1.7 195224 35728  -  IsJ  16:09   0:00.00 postgres: io worker 0 (postgres)
postgres 29997  0.0  1.7 195224 35704  -  IsJ  16:09   0:00.00 postgres: io worker 2 (postgres)
postgres 30256  0.0  1.7 195224 35704  -  IsJ  16:09   0:00.00 postgres: io worker 1 (postgres)
postgres 30624  0.0  1.7 195224 35704  -  IsJ  16:09   0:00.00 postgres: checkpointer  (postgres)
postgres 30882  0.0  1.7 195224 35704  -  SsJ  16:09   0:00.00 postgres: background writer  (postgres)
postgres 31530  0.0  1.7 195224 35752  -  SsJ  16:09   0:00.00 postgres: walwriter  (postgres)
postgres 32166  0.0  1.8 198808 36068  -  IsJ  16:09   0:00.00 postgres: autovacuum launcher  (postgres)
postgres 32730  0.0  1.8 198808 36040  -  IsJ  16:09   0:00.00 postgres: logical replication launcher  (postgres)

Bash: 
# jail -v -r postgres
postgres: run command in jail as postgres: /usr/local/bin/pg_ctl -D /var/db/postgres/data -l /var/db/postgres/logfile stop
postgres: removed
postgres: run command: /sbin/ifconfig vnet80 deletem epair11a
postgres: run command: /sbin/ifconfig epair11a destroy
postgres: run command: /sbin/umount /jail/postgres/dev
postgres: run command: /sbin/umount -t nullfs /jail/postgres/var/db/postgres
 
You must log in or register to reply here.
Back
Top