Solved VNET Jails on Un-tagged LAN and Tagged VLAN

D

Deviant0ne

I would like to create a VNET-based jail that attaches to my un-tagged LAN network via a bridge interface. I have jails that connect to tagged VLAN's and that seems to be working fine, but I can't get the correct syntax together to bring the untagged bridge online. I am trying to accomplish this using iocage and my /etc/rc.conf file.

My main, un-tagged LAN is 10.0.1.0/24 (LAGG configured) and my jail VLAN is 172.16.55.0/24 (VLAN tag 55). Here is my /etc/rc.conf file:

Code: 
###  Link aggregation of both physical NIC's
cloned_interfaces="lagg0 bridge0 bridge1"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 up"
ifconfig_lagg0_alias0="inet 10.0.1.10/24"
defaultrouter="10.0.1.1"

###  Bridge interface for LAN-attached jails
vlans_lagg0="1"
ifconfig_lagg0_1="up"
ifconfig_bridge1="addm em1 up"

###  Bridge interface for jail VLAN (55)
vlans_lagg0="55"
ifconfig_lagg0_55="up"
ifconfig_bridge0="addm lagg0.55 up"

###  iocage Jails
iocage_enable="YES"

I am pretty sure the issue is that I am tagging the LAN traffic with VLAN ID 1 and the traffic needs to be untagged, but I am having trouble understanding how to get around this. If I plumb lagg0 into bridge1, then my jails are unable to access the network/Internet/etc. (which makes sense). I was also thinking about trying to plumb em1 into bridge1 but I don't want to break my LAGG. I want to use VNET to keep everything consistent - I want all jails to have their own network stack.

For what it's worth, this is the command I was using to create the jail with iocage:
Code: 
[COLOR=#000000]iocage create -r LATEST -n lanjailtest1 interfaces="vnet0:bridge1" ip4_addr="vnet0|10.0.1.11/24" defaultrouter="10.0.1.1" vnet_default_interface="bridge1" vnet=on boot=on allow_raw_sockets=on[/COLOR]

Any assistance anyone might be able to provide would be much appreciated. Also, if there is a better way to configure the VLAN bridge I would be open to making changes to that as well. I pieced that together from iocage's documentation and other blog/forum posts. Many thanks in advance!
 
Hi. Please, read my answer about native VLAN and bridges.

Your lagg0 interface is a trunk with native VLAN's. For some reasons, when you bridge whole trunk interface with with some, say, epair interface (which is how your jails usually get them interfaces in VNET case), traffic from trunk VLAN's won't reach any other bridges. Here's an example:

em0 is a trunk interface.
vlan4 is a VLAN interface with options "vlandev em0 vlan 4".

If you create bridge4 and add vlan4 to it, all bridge4 members will receive traffic in VLAN 4.

Then, if you create bridge0 and add em0 to it, all bridge0 members will receive ALL traffic (in every VLAN) that comes through em0, and - here's a significant part - vlan4 (and so bridge4) will stop receiving VLAN 4 traffic. I don't know why is this happening, but that's how it's working.

I found a clear and simple way to overcome this: tag VLAN 1 in trunk (making native VLAN some dummy VLAN - 999 is good number for it), create interface vlan1 for it, add it to bridge1, and for every jail that need that VLAN bridge-switch, add jail's epair interface to that bridge.

Configure VLAN 999 as a native on you lagg0 trunk, and your config would be like this:

Code: 
###  Link aggregation of both physical NIC's
cloned_interfaces="lagg0 bridge1 bridge55 vlan1 vlan55"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 up"

ifconfig_vlan1="inet 10.0.1.10/24 vlandev lagg0 vlan 1 up"
ifconfig_vlan55="vlandev lagg0 vlan 55 up"

defaultrouter="10.0.1.1"

###  Bridge interface for LAN-attached jails
ifconfig_bridge1="addm vlan1 up"

###  Bridge interface for jail VLAN (55)
ifconfig_bridge55="addm vlan55 up"

###  iocage Jails
iocage_enable="YES"
 
sol289 said:
Hi. Please, read my answer about native VLAN and bridges.

Your lagg0 interface is a trunk with native VLAN's. For some reasons, when you bridge whole trunk interface with with some, say, epair interface (which is how your jails usually get them interfaces in VNET case), traffic from trunk VLAN's won't reach any other bridges. Here's an example:

em0 is a trunk interface.
vlan4 is a VLAN interface with options "vlandev em0 vlan 4".

If you create bridge4 and add vlan4 to it, all bridge4 members will receive traffic in VLAN 4.

Then, if you create bridge0 and add em0 to it, all bridge0 members will receive ALL traffic (in every VLAN) that comes through em0, and - here's a significant part - vlan4 (and so bridge4) will stop receiving VLAN 4 traffic. I don't know why is this happening, but that's how it's working.

I found a clear and simple way to overcome this: tag VLAN 1 in trunk (making native VLAN some dummy VLAN - 999 is good number for it), create interface vlan1 for it, add it to bridge1, and for every jail that need that VLAN bridge-switch, add jail's epair interface to that bridge.

Configure VLAN 999 as a native on you lagg0 trunk, and your config would be like this:

Code: 
###  Link aggregation of both physical NIC's
cloned_interfaces="lagg0 bridge1 bridge55 vlan1 vlan55"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 up"

ifconfig_vlan1="inet 10.0.1.10/24 vlandev lagg0 vlan 1 up"
ifconfig_vlan55="vlandev lagg0 vlan 55 up"

defaultrouter="10.0.1.1"

###  Bridge interface for LAN-attached jails
ifconfig_bridge1="addm vlan1 up"

###  Bridge interface for jail VLAN (55)
ifconfig_bridge55="addm vlan55 up"

###  iocage Jails
iocage_enable="YES"
Click to expand...
Thank you again for the reply. I used the configuration outlined above but on reboot, I found that vlan1 was not coming up and using direct console access to the server, I found that the server was not able to set 10.0.1.10/24 on the lagg0 adapter. As such, I was not able to communicate with the gateway. I re-enabled the alias and rebooted to regain network and Internet access.

What's strange is that there is no mention of vlan1 in /var/log/messages.

Thank you for your continued assistance and suggestions!
 
Did you change cloned_interfaces="" line of /etc/rc.conf? That's where vlan1 interface is created.

Show your /etc/rc.conf and show output of ifconfig command.
 
sol289 said:
Did you change cloned_interfaces="" line of /etc/rc.conf? That's where vlan1 interface is created.

Show your /etc/rc.conf and show output of ifconfig command.
Click to expand...
I found where my issue was, I mis-typed something in my /etc/rc.conf. However, the same issue still exists: with the exact configuration above, I am not able to access the server from the LAN and the server is not able to connect to anything on the LAN, Internet, etc. The jails running on their respective VLANs are working normally, however.

The more I read and re-read this, it seems like I might need to completely re-architect my network, i.e. change the native VLAN to something other than 1. Essentially, I would need to create a new VLAN for all of my LAN devices and migrate settings to the new interface, etc. (if I am understanding this correctly). It sounds to me like my desired configuration is maybe not possible?
 
sol289 said:
I thought you do this first. Yes, without changing native VLAN in trunk you would not be able to decouple VLAN 1 from trunk.
Click to expand...
Rather than change the default VLAN for my LAN network, I opted to create yet another VLAN and configure the jail I wanted there. I needed Avahi to advertise services, so I configured the Avahi package in my firewall and now everything is working as expected.

This issue is eating at me, I am still going to look for ways around this limitation. I was looking into creating a VALE switch, vale(4), valectl(8), but the documentation is not terribly clear and there don't seem to be many examples to learn from. I was also thinking about installing another NIC and dedicating that NIC to LAN jails.

I can't be the only one with this issue, right?
 
Deviant0ne said:
I can't be the only one with this issue, right?
Click to expand...
You mean, issue about bridging native VLAN? Yes, we are all here with this issue. I don't see this as a problem, rather than this is how FreeBSD network stack is built. I'm suffering more of bridge throughtput, but it's seems that issue is fixed in CURRENT.

Do you REALLY need virtual switch?
 
sol289 said:
You mean, issue about bridging native VLAN? Yes, we are all here with this issue. I don't see this as a problem, rather than this is how FreeBSD network stack is built. I'm suffering more of bridge throughtput, but it's seems that issue is fixed in CURRENT.

Do you REALLY need virtual switch?
Click to expand...
WANT and NEED are subjective terms, I suppose. I would really like to have all jails use VNET instead of attaching one or two jails to existing host adapter(s). I think mostly, I want this for consistency/my own OCD - I also don't love that I have to run extra services on my firewall to accomplish my desired outcome.

I am also looking into VXLAN's to see if that will bring me any closer.
 
I'm running jails using VNET with one host adapter, there is one trunk interface with native VLAN set to dummy VLAN, and jails that connects to one or more VLANs with epairs and bridges, and some of them connects to VLAN 1, without any problems. Yes, most of work is done by hands, and network configuration is mixed in jails configuration, but is there an alternative?
 
sol289 said:
I'm running jails using VNET with one host adapter, there is one trunk interface with native VLAN set to dummy VLAN, and jails that connects to one or more VLANs with epairs and bridges, and some of them connects to VLAN 1, without any problems. Yes, most of work is done by hands, and network configuration is mixed in jails configuration, but is there an alternative?
Click to expand...
I am still curious about your configuration. I don't fully understand why you needed to move your native/default/untagged VLAN to another ID in the first place. Couldn't you have just used VLAN 999 for your dummy VLAN without needing to adjust the default settings in your switch(es) and firewall? Why did you use VLAN 1 at all? Also, since your network traffic is being tagged with VLAN ID 1 and your default VLAN is now 999, why isn't your switch dropping the traffic?
 
"My" network is a small ISP, there is a specific IP network in this VLAN 1 with many hosts, and only one of them is FreeBSD with jails, on which I need that VLAN 1 to be in a jail. And through this trunk also comes in all other VLANs, which ends up in other jails on this host. And there are even 1 VM with Linux on this FreeBSD server. And so it's simpler to change trunk configuration on switch than reconfigure VLAN 1 all the way through the not-so-big, but not-so-small network.
 
sol289 said:
"My" network is a small ISP, there is a specific IP network in this VLAN 1 with many hosts, and only one of them is FreeBSD with jails, on which I need that VLAN 1 to be in a jail. And through this trunk also comes in all other VLANs, which ends up in other jails on this host. And there are even 1 VM with Linux on this FreeBSD server. And so it's simpler to change trunk configuration on switch than reconfigure VLAN 1 all the way through the not-so-big, but not-so-small network.
Click to expand...
Thanks for your input on this - I was able to get a VNET jail attached to my LAN finally. I needed to use ngctl from netgraph to create a bridge interface, attach my LAGG to the ng_bridge interface and then create VNET interfaces attached to the bridge.

Also, I have only been able to get the jail created using /etc/jail.conf, I haven't been successful at using iocage to create the jail.
 
You're welcome! And thanks for sharing your scheme. Could you please show your NETGRAPH tree? And your jail's configuration? How do you decouple VLANs inside of jail?

I manage jails with /etc/jail.conf myself, and I think it's a clear and powerful way to manage jails like this.
 
sol289 said:
You're welcome! And thanks for sharing your scheme. Could you please show your NETGRAPH tree? And your jail's configuration? How do you decouple VLANs inside of jail?

I manage jails with /etc/jail.conf myself, and I think it's a clear and powerful way to manage jails like this.
Click to expand...
After I got this working manually, I found someone else that scripted the process.

genneko’s Learning Notes on FreeBSD Jails

Update 20210103 - I tested my configuration using simply the /usr/share/examples/jails/jng script and found that I was able to achieve my desired outcome.
 
Last edited:
You must log in or register to reply here.
Back
Top