jails Nested Jails suffer from leaky abstractions : jail$ service netif restart

toddg

toddg

I've working on a custom jail deployment system, glm_freebsd_cloud, to create nested jails.

SUMMARY
My goal is to have isolated environments that don't destroy each other, meaning they have a contained blast radius.

Example:
Bash: 
HOST -> PARENT_JAIL -> CHILD_JAIL
...
HOST -> PARENT_JAIL_DEV -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_DEV -> CHILD_JAIL_DB

HOST -> PARENT_JAIL_STAGE -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_STAGE -> CHILD_JAIL_DB

HOST -> PARENT_JAIL_PROD -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_PROD -> CHILD_JAIL_DB

I _thought_ that nested jails would be perfect for this. However, nested jails can destroy the entire stack by restarting the jails network:

Bash: 
jail$ service netif restart

This destroys the jails network interface, just as if you'd run:

Bash: 
jail$ ifconfig [jails_epair_b] destroy

There is no way to re-create the epair from within the jail. I'm curious why the JAIL can destroy it's network interface if it cannot re-create it... can a HOST destroy it's network interface and not be able to recreate it?
The kicker is that restarting the network from within the jail also completely screws up the host network, in ways that I don't understand, but can see the effects of. I end up re-deploying all my jails at the HOST level when this used to happen. I've replaced my code that restarted the netif within the jails with a script (below).

DETAILS
Here's my workflow:
1. create a deployment, consisting of HOST (level0) files: /etc/rc.conf, /etc/pf.conf, /etc/jail.conf.d/jail_conf_files...
2. deploy those files to the HOST and restart or reload services as necessary
3. jexec into a jail that was created in #1 and #2 above, and turn it into a parent jail by doing #1 and #2 above, but in the jail

PROBLEM
In the HOST, because I can restart networking, I can basically get the stuff I need from /etc/rc.conf to be reloaded, e.g. networking (interfaces, gateway_enable, routing_enable, cloned interfaces, bridge definitions, bridge aliases, etc.)
However, in the JAIL, I cannot restart networking without breaking stuff, so I had to create a script to configure networking. I think this is a bad idea (tm), because now I'm tracking configuration in multiple places, the apply_bridge_config.script, and in /etc/rc.conf.

SOLUTIONS
I'd like for the leaky abstractions to be closed down such that a JAIL really is a self contained OS.

In the meantime, if anyone has any better ideas than what I'm doing, I'm all ears...

FILES
For the curious, here's a full set of generated file for both the host and jails:
* host files
* jail files
* bodge script that I'm using to work around not restarting netif in jails
 
codeedog pprocacci patmaddox

I'm working on getting nested jails working. I gave up on getting vlans to work, and just reverted to using a single bridge with fine grained filtering in pf.conf. This is working well for level0.

In level0, host and immediate jails, networking functions as expected.

However, in level1, the child jails are not able to talk to the outside world. I only see ping requests from the children reaching the parent.

Do any of you have a working example of nested jails that you could share?
 
I figured out the issue(s) after some quality time with tcpdump.

I'll post all my working files once I've updated my code generators, test cases, etc.

0. network topology

Bash: 
HOST                       -> PARENT_JAIL     -> CHILD_JAIL
192.168.0.40/32  -> 10.0.1.0/24           -> 10.1.0.0/24

1. parent jail was not forwarding packets

This caught me up b/c, as I mentioned earlier, I provision jail hosts in two steps. First, I provision the level0 resources, e.g. parent jails. Then, I enter into the parent jail and provision it's resources, e.g. the leaf jails (I'm not going any deeper than that). However, even though the parent jail has 'gateway_enabled="YES"'... this is added after the jail has been created. The only way to reload this updated rc.conf file is to restart the jail (remember restarting netif from within the jail will destroy the epairs created in level0, breaking level0 networking in various ways). So, I updated my hack, 'append_bridge_config.sh'. Remember, this get's run as part of a deployment, and get's around the issue of rc.conf not getting reloaded.

Bash: 
 # -----------------------------------------------------------------------
-log "Step 9: Final verification"
+log "Step 9: Update Jail Host"
+# -----------------------------------------------------------------------
+sysctl net.inet.ip.forwarding=1

2. no need to NAT from the level1 parent host

Some LLM told me to NAT from level1 to level0. But this was dumb. The same LLM later told me so. Hah. Anyway, I eventually figured out that this was, in fact, incorrect and my original idea to just let the 10.1.0.0/24 packets onto the parent jail's bridge would work fine, so long as I updated the HOST pf.conf to correctly NAT the 10.1.0.0/24 packets. This stanza in my pf.conf template only enables NAT for level0, e.g. host pf.conf files:

Bash: 
 # --------------------------------------------------------------
 # Translations (NAT)
 # --------------------------------------------------------------
+{{#if server_config.jail_host_config.is_nat_enabled}}
 nat on $ext_if inet from <jails_v4> to any -> ($ext_if:0)
+{{/if}}


3. host bridge could not route return packets to the child jail

Cool, so now I have packets from my child jail making an almost round trip, but return packets don't make it off the host bridge:

Bash: 
# outbound
child jail (epair_b) -> parent jail (child jail epair_a) -> parent jail bridge -> parent jail (epair_b) -> host (parent jail epair_a) -> host bridge -> internet
# inbound
internet -> host bridge -> ...nothing

BTW - I diagnosed all of this by opening a bunch of terminals, one on each of the nodes listed above, and running something like:

Bash: 
tcpdump -nn -tttt -v -i [interface name] 'net 10.1.0.0/24'

The root issue is that the packets are coming back looking for a path to 10.1.0.4, but the bridge didn't have an alias for 10.1.0.0/24...so there was no where to go. They were all silently dropped.

** still working on the fix **

I also have some nagging issues with my parent jail pf.conf file not properly restricting child jail egress traffic. Once I solve these, I'll post my full set of configs here.
 
You must log in or register to reply here.
Back
Top