jails Thoughts on netif configuration of epairs & bridge for VNET jails

[codeedog]

For the past day, I've been fighting with configuring three jails, their epairs, a bridge and the host network interfaces. Things work now, but I feel it's worthwhile detailing my findings. The upshot is that I found it impossible to configure the network in such a way that upon reboot or service netif restart, everything came up and connected. I could manufacture the network from scratch at the command line and I could coax the network to life after reboot by bringing up devices and adding them to the bridge, but I could not reliably configure the network in /etc/rc.conf or /etc/rc.conf.d/netif.

I solved my problem by creating a cleanup service (netif-cleanup) which requires NETWORKING and finishes what netif should have done. I have to say, I really appreciate the service methodology as a means of handling FreeBSD's systems. Putting together the clean up code was very simple.

Anyway, I realize most people just use Devin Teske's jib, which works well. However, it doesn't work with child jails and has some other side-effects, and I'm in a learning mode and want to use as few tools as possible.

My set up is three jails: one handling DNS and DHCP, and the other two relying on the first jail. Each jail has an associated epair the other end of which goes into a bridge. There are two physical NICs, one for WAN one for a trunked LAN. The latter is also added to the bridge. I'm building a gateway/firewall.

As I said, everything works well once all interfaces are up and connected through the bridge.

The problem I had was that upon reboot or restart, only the LAN interface was in the bridge and some of the epair ends were down. It didn't matter how I ordered the configuration data, something always wasn't right. I should note that I was also renaming the epairs and am unsure about that impact.

I can't tell if netif reasons about when to add an interface to a bridge, if it has some generic buckets for creation, configuration, connection and then whatever happens happens. I'm not sure what more I could have done to get the configuration correct.

 

[mumu]

This might/might not be related but for starting and stopping services, I am using just 2 scripts to handle this, /etc/rc.local and /etc/rc.shutdown.

I went this route as I found out certain services would hang on boot and stays that way till a force reboot is applied.

/etc/rc.local is ran after the system has booted, and the services to start in that script are in particular order.

/etc/rc.shutdown does the opposite and turns off services via sysrc so they won't start during boot.

This can apply to networking as well, for say epair's, etc..

 

[victort]

I’ve resolved to always manually create a bridge, and attach all my jails to it. It works better than the JIB script IMO.

I use bastille for my jail management.

 

[codeedog]

This might/might not be related but for starting and stopping services, I am using just 2 scripts to handle this, /etc/rc.local and /etc/rc.shutdown.
<snip>

I took a look at the /etc/rc.local option, and felt it didn't address my issue.

The problem is that

service netif restart

and boot should both work the same way *and* there should be a way to convey to netif the software defined network created in the configuration files. Unless netif does the analysis and builds a network graph (this interface has been created, renamed and placed in this renamed bridge, for example), it cannot stably bring up real, vlan, and cloned devices and connect them together. So, something is always down or left out or both.

The jib program adds to this complexity, but I tried it without the jib program and still ran into a problem. Perhaps, there's a way to get it to always work on reboot (I'm not sure), but to me it seams unreasonable to have to reboot a gateway just to make sure it's network configuration is correct. That's why the service restart code is there.

I'm guessing netgraph doesn't suffer from this problem as creation is scripted and explicit. However, I'm not going to adopt a new network stack at this time. That's a future project.

Thanks for the suggestion!

 

[codeedog]

I’ve resolved to always manually create a bridge, and attach all my jails to it. It works better than the JIB script IMO.

I use bastille for my jail management.

This is a good idea and I tried manual creation.

The challenge is that I have a number of VLANs, so the combination of VLANxBRIDGExJAIL makes for some polymorphic expansion of cloned interfaces and jib assists with reducing things. Especially, when you consider that an epair has to be created for each jail that attaches to a bridge. If you add a trunked interface (with VLANs) to a bridge and then attach an epair, the epair does not "see" any of the VLANs. Effectively, you have to fan out each VLAN from the trunk interface into a separate bridge.

I have six VLANs on the trunk, so I have six different bridges with at least one trunk VLAN interface and at least one epair end connected to it. Then, the other epair end goes to a jail. So, that's four interfaces times six VLANs or 24 interfaces. It's screaming for automation.

If jib didn't exist, I'd have to invent it.

Also, thanks for the suggestion re: Bastille. I've looked at the jail managers and am considering using one, although not sure which one at this time. I'm not entirely convinced they'd help with this problem as I think it's the VLANs that throw things for a loop. Also, see my other comment for why netif has some instabilities, which I do not believe that a jail manager could fix, although perhaps they do?

Finally, I want as little tooling as possible, right now, as I want to learn everything and get my hands dirty and I feel like the tools keep me ignorant of the lower, raw layers. I tried to do this without jib, btw and you can see it started to get ugly (who wants to manage 24 interfaces?). I'm comfortable enough with ifconfig and netif now that I don't feel like I'm missing out by using jib on top of them.

 

[mumu]

This is a good idea and I tried manual creation.

The challenge is that I have a number of VLANs, so the combination of VLANxBRIDGExJAIL makes for some polymorphic expansion of cloned interfaces and jib assists with reducing things. Especially, when you consider that an epair has to be created for each jail that attaches to a bridge. If you add a trunked interface (with VLANs) to a bridge and then attach an epair, the epair does not "see" any of the VLANs. Effectively, you have to fan out each VLAN from the trunk interface into a separate bridge.

I have six VLANs on the trunk, so I have six different bridges with at least one trunk VLAN interface and at least one epair end connected to it. Then, the other epair end goes to a jail. So, that's four interfaces times six VLANs or 24 interfaces. It's screaming for automation.

If jib didn't exist, I'd have to invent it.

Also, thanks for the suggestion re: Bastille. I've looked at the jail managers and am considering using one, although not sure which one at this time. I'm not entirely convinced they'd help with this problem as I think it's the VLANs that throw things for a loop. Also, see my other comment for why netif has some instabilities, which I do not believe that a jail manager could fix, although perhaps they do?

Finally, I want as little tooling as possible, right now, as I want to learn everything and get my hands dirty and I feel like the tools keep me ignorant of the lower, raw layers. I tried to do this without jib, btw and you can see it started to get ugly (who wants to manage 24 interfaces?). I'm comfortable enough with ifconfig and netif now that I don't feel like I'm missing out by using jib on top of them.

What if you don't rename your interfaces and use the default names. Does netif restart still not bring up and configure all interfaces ?

 

[codeedog]

What if you don't rename your interfaces and use the default names. Does netif restart still not bring up and configure all interfaces ?

IIRC, yes, it still fails to bring them up in the correct configuration—bridges are definitely missing their addm.

FWIW, if it did work non-renamed, then I'd have to live with interfaces named "bridge6" and "epair8a". That'd be a situation fraught with peril. I'm much better off renaming the interfaces and just running my netif-cleanup service to knit them together.