Multiple jails with unique IP addresses

E

Elliott

I'm interested in setting up a few jails on a small home server. From what I understand, there are two ways to go about this.
  1. Assign each jail a unique IP address on the same subnet as my home network. Use port forwarding on my router to route traffic to each one.
  2. Assign each jail a unique IP on a private subnet. Use pf to route traffic internally.

What are the pros and cons of these options? My gut tells me to use the first option because 2 NATs would add latency and maybe increase CPU / power consumption a tiny bit. Currently I am not running pf at all.
 
On a similar note, what is the purpose of assigning two IP addresses to one jail? I followed the instructions here:

It says the purpose is
"To keep jail loopback traffic off the host's loopback network interface lo0"
Click to expand...
But then it also says
"Inside a jail, access to the loopback address 127.0.0.1 is redirected to the first IP address assigned to the jail"
Click to expand...
The example command is

ezjail-admin create dnsjail 'lo1|127.0.1.1,em0|192.168.1.50'


Why not just assign one IP address 192.168.1.50 ? Would this not achieve the same thing? This is actually causing a problem for me because Plex Media Server binds to the first interface.
 
A "traditional" jail shares the network stack with the host and all other jails. This means the only separation happens on the "IP" level, you can restrict a jail to specific IP addresses. As all of these addresses are created on interfaces of the host system, the host system sees them all, and you have to be careful to configure any network services to specifically listen on the IP addresses you want.

That said, I think you misunderstood something when you write "2 NATs". If you want to expose a service in a jail to the outside via a public IP address, you need to do NAT exactly once, no matter whether your jails are all sharing a subnet or have their own. The difference is that with separate subnets, you are in control of the routing between jails (and the host), so you could apply firewall rules there as well.

For some time now, you can go one step further and use VNET jails, which means the jail has its own private network stack -- you typically give it an epair(4) interface that's completely separate from the host's interfaces. The other end of the epair can be routed (like giving each jail its own subnet) or bridged (somewhat similar to have all jails share a common subnet). IMHO, this concept is easier to grasp and gives an even better separation, so I personally recommend using VNET jails.
 
I'm not sure if ezjail can do VNETs. It sounds like I need to abandon that and read up on jails.conf.

Regarding the 2 NATs, my home has one public IP address, NATted by the router to 192.168.1.0/24. In my first example using pf, each jail is in the 127.0.1.0/24 subnet, so I would access the server at 192.168.1.5 and this would translate to 127.0.1.1. Isn't that considered double NAT?
 
I recommand learning / experimenting jails without ezjail/iocage/bastille...
Because once you know what FreeBSD jails is, you understood all this tools.
A manager tool is useless before the «industrialization» step.
 
Hakaba said:
I recommand learning / experimenting jails without ezjail/iocage/bastille...
Because once you know what FreeBSD jails is, you understood all this tools.
A manager tool is useless before the «industrialization» step.
Click to expand...
+ 1 for that ... managing jails with the tools the base system is giving you is very straight forward! In addition to that, iocage cost me hours for its bugs (I call it bugs, the author calls it design decisions), my tipp: stay away from iocage especially if you plan on using VNET jails, go with the systems standard tools!
 
Yes, I generally prefer to do things manually for better understanding, at least to start with.

I'm trying to start simple, and so far I have not had any need for downloading source code. I'm installing packages, not ports. But all the documentation I read about jails uses something like this

D=/here/is/the/jail
cd /usr/src
mkdir -p $D
make world DESTDIR=$D
make distribution DESTDIR=$D

Is it required to build the jail from source or is there a faster way?
 
My use case is similar to yours, and iocage works well for me.

It looks like there is a Plex Media Server plugin for iocage, which should simplify configuration if you decide to try it. I have never used it.

I see things the other way; use iocage to get a test jail working, then examine the files it produces to understand what the underlying system is doing.
 
To answer OP's question, Option 1 (what I do with iocage) means all ports are accessible within your home network. This is simpler as there is less moving components, but every port on your jails is accessible to your whole network unless you firewall on the jail itself.

Option 2 (used by Docker) saves IPv4 addresses. This is useful on cloud providers with 1 public IP/VM but not as much your home LAN which is already behind NAT. This has the disadvantage of more complexity ( pf/ ipfw needed).

If you use Jails on a home network, I'd recommend Option 1. Less moving components and it's not like your LAN is running out of internal addresses. And NAT is heavier than just pure firewalling if you need it.
 
Elliott said:
Yes, I generally prefer to do things manually for better understanding, at least to start with.

I'm trying to start simple, and so far I have not had any need for downloading source code. I'm installing packages, not ports. But all the documentation I read about jails uses something like this

D=/here/is/the/jail
cd /usr/src
mkdir -p $D
make world DESTDIR=$D
make distribution DESTDIR=$D

Is it required to build the jail from source or is there a faster way?
Click to expand...

You can do this:
mkdir /jails/new_jail
bsdinstall jail /jails/new_jail
freebsd-update -b /jails/new_jail fetch install
 
I tried initially with pure jails, but gave up for the time being, too many unknowns and not enough time. Eventually got it working with iocage + vnet. I have not used it that much to get into any bugs or major issues. I like the concept of releases (don't have to manually create the first jail) and templates. Easy to work and automate (for me at least). I have a base R12.1 template with my ssh key, which I clone and assign a separate IP. After that I use ansible to finish setup.

I could not figure out a simple way to automate updating /etc/jail.conf with new jails, because it's a single file. I wish there were includes and/or separate files.

This is my script to create new jails
Code: 
#!/bin/sh

JAIL_TEMPLATE=$1
JAIL_NAME=$2
JAIL_IP=$3

iocage create -T -t $JAIL_TEMPLATE -n $JAIL_NAME
iocage set allow_raw_sockets=1 vnet=on ip6_addr=none defaultrouter6=none defaultrouter=192.168.1.1 ip4_addr="vnet0|$JAIL_IP/24" $JAIL_NAME
iocage start $JAIL_NAME

Btw, this is how iocage uses freebsd-update to upgrade jails (saw that while having htop running).
Code: 
/bin/sh /tmp/tmp3cn1hmz9 -b /z/iocage/jails/test1/root -d /z/iocage/jails/test1/root/var/db/freebsd-update/ -f /z/iocage/jails/test1/root/etc/freebsd-update.conf --not-running-from-cron fetch

I had an issue with iocage pkg though, it was either super slow or froze, but it's not a blocker I can use ansible and pkgng
 
You must log in or register to reply here.
Back
Top