I've working on a custom jail deployment system, glm_freebsd_cloud, to create nested jails.
SUMMARY
My goal is to have isolated environments that don't destroy each other, meaning they have a contained blast radius.
Example:
I _thought_ that nested jails would be perfect for this. However, nested jails can destroy the entire stack by restarting the jails network:
This destroys the jails network interface, just as if you'd run:
There is no way to re-create the epair from within the jail. I'm curious why the JAIL can destroy it's network interface if it cannot re-create it... can a HOST destroy it's network interface and not be able to recreate it?
The kicker is that restarting the network from within the jail also completely screws up the host network, in ways that I don't understand, but can see the effects of. I end up re-deploying all my jails at the HOST level when this used to happen. I've replaced my code that restarted the netif within the jails with a script (below).
DETAILS
Here's my workflow:
1. create a deployment, consisting of HOST (level0) files: /etc/rc.conf, /etc/pf.conf, /etc/jail.conf.d/jail_conf_files...
2. deploy those files to the HOST and restart or reload services as necessary
3. jexec into a jail that was created in #1 and #2 above, and turn it into a parent jail by doing #1 and #2 above, but in the jail
PROBLEM
In the HOST, because I can restart networking, I can basically get the stuff I need from /etc/rc.conf to be reloaded, e.g. networking (interfaces, gateway_enable, routing_enable, cloned interfaces, bridge definitions, bridge aliases, etc.)
However, in the JAIL, I cannot restart networking without breaking stuff, so I had to create a script to configure networking. I think this is a bad idea (tm), because now I'm tracking configuration in multiple places, the apply_bridge_config.script, and in /etc/rc.conf.
SOLUTIONS
I'd like for the leaky abstractions to be closed down such that a JAIL really is a self contained OS.
In the meantime, if anyone has any better ideas than what I'm doing, I'm all ears...
FILES
For the curious, here's a full set of generated file for both the host and jails:
* host files
* jail files
* bodge script that I'm using to work around not restarting netif in jails
SUMMARY
My goal is to have isolated environments that don't destroy each other, meaning they have a contained blast radius.
Example:
Bash:
HOST -> PARENT_JAIL -> CHILD_JAIL
...
HOST -> PARENT_JAIL_DEV -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_DEV -> CHILD_JAIL_DB
HOST -> PARENT_JAIL_STAGE -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_STAGE -> CHILD_JAIL_DB
HOST -> PARENT_JAIL_PROD -> CHILD_JAIL_WEB
HOST -> PARENT_JAIL_PROD -> CHILD_JAIL_DBI _thought_ that nested jails would be perfect for this. However, nested jails can destroy the entire stack by restarting the jails network:
Bash:
jail$ service netif restartThis destroys the jails network interface, just as if you'd run:
Bash:
jail$ ifconfig [jails_epair_b] destroyThere is no way to re-create the epair from within the jail. I'm curious why the JAIL can destroy it's network interface if it cannot re-create it... can a HOST destroy it's network interface and not be able to recreate it?
The kicker is that restarting the network from within the jail also completely screws up the host network, in ways that I don't understand, but can see the effects of. I end up re-deploying all my jails at the HOST level when this used to happen. I've replaced my code that restarted the netif within the jails with a script (below).
DETAILS
Here's my workflow:
1. create a deployment, consisting of HOST (level0) files: /etc/rc.conf, /etc/pf.conf, /etc/jail.conf.d/jail_conf_files...
2. deploy those files to the HOST and restart or reload services as necessary
3. jexec into a jail that was created in #1 and #2 above, and turn it into a parent jail by doing #1 and #2 above, but in the jail
PROBLEM
In the HOST, because I can restart networking, I can basically get the stuff I need from /etc/rc.conf to be reloaded, e.g. networking (interfaces, gateway_enable, routing_enable, cloned interfaces, bridge definitions, bridge aliases, etc.)
However, in the JAIL, I cannot restart networking without breaking stuff, so I had to create a script to configure networking. I think this is a bad idea (tm), because now I'm tracking configuration in multiple places, the apply_bridge_config.script, and in /etc/rc.conf.
SOLUTIONS
I'd like for the leaky abstractions to be closed down such that a JAIL really is a self contained OS.
In the meantime, if anyone has any better ideas than what I'm doing, I'm all ears...
FILES
For the curious, here's a full set of generated file for both the host and jails:
* host files
* jail files
* bodge script that I'm using to work around not restarting netif in jails