Intro:
It appears ~Luna in [this] post and I are trying to do something very similar. Instead of hijacking the original thread, I decided to create a new one (but with the link to the original question by ~Luna, maybe it will help them).
Current State:
I am currently running a handful of applications in jails and because there is a handful of them, I was able to "get by" cloning a lo1 on the host, assigning a predefined ip4 range and using IPs from this range in jail definitions (really, hardcoding them in the jail configuration file). I am also using pf for nat traffic in and out of jails where this is necessary. The cloning of lo1 interface and hardcoding ipv4 addresses in jails appears to be the solution to how lo0 interface behaves in jails (it is done to basically prevent jailed services from binding to the host's loopback interface).
Desired State:
Minimally, I want to run a set of jails (let's say, 100 different services) on their own subnet with ip addresses handed over by the dhcpd service running on one of the jails on this subnet. Obviously, manually hardcoding 100 ip addresses in the jail configuration files is not sustainable anymore, so definitely need dhcp paired with bind for name resolution.
So far I tried:
1. VNET jails with NetGraph (I ruled out epair bridging because this leaves the a-pair visible on the host, and I would eventfully end up seeing 100s of interfaces). I used the `jng` script to create a netgraph (ng) bridge with the host's "real" interface and create any additional eifaces for individual jails. This *almost* worked: I was able to give static ip addresses to vnet jails if I wanted to, and once the dhcpd service was running in the jail, it was happy to give out ip addresses to any other jails with a dynamic ip configuration. The almost part is because I realized that the dhcp server running in a jail was also responding to DHCP requests made by the devices outside of my jail subnet. I believe this is a side effect of using the *bridge* which allows the broadcast traffic to pass in either direction its ethernet neighbors. The stumbling block is that I can't figure out the pf rule to block the broadcast traffic from leaving the net graph bridge, especially, since it may not be visible to it (the net graph bridge does not show up in the output of ifconfig cmd on the host). I can write a rule to block/allow traffic by using subnet criteria, but to write a rule without having the iface is beyond by skill level
.
2. This led me to the next solution: created a new net graph eiface device for the host is bridged it with a "real" iface using net graph's bridge. The idea is to use this new iface (ngeth0) in pf rules to block traffic at the interface level. So, the device creation part was not trivial, but I was able to do it yesterday. However, when I tried to use this new eiface (ngeth0) in the `jng` script, it didn't work. After some troubleshooting it appeared to have failed in enabling the promiscuous mode on the "fake" eiface ngeth0 (or maybe it was the new ng bridge - can't remember now)
3. This whole thing may be easier if I were to just add another network interface to the host (it just occurred to me yesterday, doh!!!), and use one of jails, and one for everything else; the whole thing is running on the bhive virtual machine, anyways, but I can't help thinking what would happen when I need to deploy it on a real box (dedicate a set of network cards for jails? is this what people do?). I didn't try this one yet, but don't see why this wouldn't work.
Still, the open questions:
1. (Probably simple) Can I use a pf rule to block broadcast traffic coming from the bridge connected to the network interface (without blocking the broadcast traffic on the network interface itself)? I prefer pf since I already invested a lot of time into educating myself in syntax, etc.
2. (Open-ended) Did anyone (other than ~Luna) attempt a similar setup and care to share experience?
It appears ~Luna in [this] post and I are trying to do something very similar. Instead of hijacking the original thread, I decided to create a new one (but with the link to the original question by ~Luna, maybe it will help them).
Current State:
I am currently running a handful of applications in jails and because there is a handful of them, I was able to "get by" cloning a lo1 on the host, assigning a predefined ip4 range and using IPs from this range in jail definitions (really, hardcoding them in the jail configuration file). I am also using pf for nat traffic in and out of jails where this is necessary. The cloning of lo1 interface and hardcoding ipv4 addresses in jails appears to be the solution to how lo0 interface behaves in jails (it is done to basically prevent jailed services from binding to the host's loopback interface).
Desired State:
Minimally, I want to run a set of jails (let's say, 100 different services) on their own subnet with ip addresses handed over by the dhcpd service running on one of the jails on this subnet. Obviously, manually hardcoding 100 ip addresses in the jail configuration files is not sustainable anymore, so definitely need dhcp paired with bind for name resolution.
So far I tried:
1. VNET jails with NetGraph (I ruled out epair bridging because this leaves the a-pair visible on the host, and I would eventfully end up seeing 100s of interfaces). I used the `jng` script to create a netgraph (ng) bridge with the host's "real" interface and create any additional eifaces for individual jails. This *almost* worked: I was able to give static ip addresses to vnet jails if I wanted to, and once the dhcpd service was running in the jail, it was happy to give out ip addresses to any other jails with a dynamic ip configuration. The almost part is because I realized that the dhcp server running in a jail was also responding to DHCP requests made by the devices outside of my jail subnet. I believe this is a side effect of using the *bridge* which allows the broadcast traffic to pass in either direction its ethernet neighbors. The stumbling block is that I can't figure out the pf rule to block the broadcast traffic from leaving the net graph bridge, especially, since it may not be visible to it (the net graph bridge does not show up in the output of ifconfig cmd on the host). I can write a rule to block/allow traffic by using subnet criteria, but to write a rule without having the iface is beyond by skill level
. 2. This led me to the next solution: created a new net graph eiface device for the host is bridged it with a "real" iface using net graph's bridge. The idea is to use this new iface (ngeth0) in pf rules to block traffic at the interface level. So, the device creation part was not trivial, but I was able to do it yesterday. However, when I tried to use this new eiface (ngeth0) in the `jng` script, it didn't work. After some troubleshooting it appeared to have failed in enabling the promiscuous mode on the "fake" eiface ngeth0 (or maybe it was the new ng bridge - can't remember now)
3. This whole thing may be easier if I were to just add another network interface to the host (it just occurred to me yesterday, doh!!!), and use one of jails, and one for everything else; the whole thing is running on the bhive virtual machine, anyways, but I can't help thinking what would happen when I need to deploy it on a real box (dedicate a set of network cards for jails? is this what people do?). I didn't try this one yet, but don't see why this wouldn't work.
Still, the open questions:
1. (Probably simple) Can I use a pf rule to block broadcast traffic coming from the bridge connected to the network interface (without blocking the broadcast traffic on the network interface itself)? I prefer pf since I already invested a lot of time into educating myself in syntax, etc.
2. (Open-ended) Did anyone (other than ~Luna) attempt a similar setup and care to share experience?
