networking with jails + bhyve


I am trying go setup a FreeBSD 11.1 server in a way our old Linux Server behaves.
We have an interface with an external ip-address and an internal bridge where all the containers and KVMs are connected (acting like a switch). http-port from the external ip-address is passed through to one container with NAT, and this container (nginx) does the routing of locations and subdomains which reside on other containers or KVMs. Thats also the setup which works for us with OpenBSD/vmd and virtual machines.

So far, we had a great joy setting up jails with ezjail and everything works as expected. There is a bridge0 with a tap0 (with the hosts ip-address) and a tap1 (with all jails + ip-addresses attached to).

However, we struggle with getting bhyve to run with a setup like this. I have tried bare bhyve, vm-bhyve and iohyve. With iohyve we managed to get a connection if e.g. we have a VM which is connected to tap10 (inside bridge1) and the hosts sets the ip-address (we tried to seperate jails from VMs and used the network) to on this interface. But setting up stuff like this on 10+ VMs seems too dirty. If we put tap11 to bridge1 and set an appropriate ip-address (like the setup with jails) it does not work. (ifconfig states "no carrier" which is not a problem for the jails setup)

We searched the internet, stumbled upon VIMAGE, but read that it's not stable yet producing kernel panics so this is not a choice for us. I might be on the complete wrong path and my understanding how bridges/tap-interfaces work under FreeBSD, so any enlightenment would be highly appreciated.
  • Thanks
Reactions: Oko
I was again sitting too long to solve this problem ;-) obviously, but I guess many of you know such problems. The solution however was quite simple: my mistake was giving the tap-interface the desired internal ip-address. Assigning it to bridge0 did the job. Now I have all jails + vms connected to my bridge, the jails using alltogether one tap interface and the vms using one tap interface each; bridge0 has the ip-address which is defined as gateway in the jails and vms. One sidenote for those who stumble upon this post:
Using pf and rdr to map an external port to a port on a jail is done with e.g.
rdr pass proto tcp from any to $ip_ext port 14022 -> port 22
However, this is not enough for vms where you need two rules
rdr pass proto tcp from any to $ip_ext port 15022 -> port 22
pass out quick on bridge0 proto tcp from any to port 22