Solved Bridge/epair not passing through tagged VLAN traffic between host and VNET jail

Sfynx

Sfynx

I have worked with FreeBSD and/or VLANs for quite a while now, but now I have something I cannot figure out. Hope someone can tell me what I am doing wrong.

I want to set up a FreeBSD box with VNET enabled jails where I can put each jail into a specific VLAN, all accessible via a single physical interface connected to a VLAN aware switch. For this, I have created a bridgeX interface for each VLAN. On each bridge I then put a single VLAN host interface and one or more epair interfaces, in other words, quite the standard recipe for VNET based jails. By the way, I was inspired by this guide: https://gist.github.com/sdebnath/086874c5df8b68e0df69

I guess one image says more than a thousand words in this case, this is my setup:

6761


Now onto why I wrote this forum post: it seems I cannot connect between the host and the jails this way. I cannot ping 192.168.5.51 from the host, and I cannot ping to the host (192.168.5.2) from either the 192.168.5.51 or 192.168.5.52 jails. Traffic from outside the box will also not reach the jails and vice versa.

What does work:
  • Pinging the 192.168.5.2 and 192.168.6.2 host IP addresses from outside the box when being in the respective VLANs. This indicates that the switch trunk port is configured correctly and the VLAN interfaces on the host are accepting the tagged packets.
  • Pinging between jails connected to the same bridge, e.g. pinging 192.168.5.52 from 192.168.5.51. So the bridges and epairs are forwarding packets as long as they do not go to the host.
What does not work:
  • Pinging from the host to a jail, or from a jail to the host. For instance, I cannot send anything between 192.168.5.2 and 192.168.5.51 or the other way around.
  • Pinging a jail IP from outside the box when connected to the correct VLAN. I can only ping the host itself.
In other words: it seems that something (bridge? epair?) is blocking communication between host and jails, allowing only traffic from the network to the host, or traffic between jails. I do not have any firewall such as pf enabled yet, I'd rather get this working before making it more complex by filtering any packets. Enabling IP forwarding does not fix anything and IMO would not be needed for this anyway, as in my view this should still be layer 2 material.

I have included the ifconfig and routing table information of the host and all jail environments below, I do not see anything unexpected here, but maybe you can. So there it is, can someone point me to the obvious thing I forgot so I can go on and slam my head into my desk? ;)

Used version: 12.0-RELEASE-p8

Host:
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
    ether 08:00:27:6d:e1:73
    inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
    ether 08:00:27:de:23:b6
    inet 192.168.10.101 netmask 0xffffff00 broadcast 192.168.10.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e3:55:df:8c:05
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair52a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: epair51a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: vlan5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=1<RXCSUM>
    ether 08:00:27:de:23:b6
    inet 192.168.5.2 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: em1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e3:55:df:8c:06
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair62a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: epair61a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: vlan6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=1<RXCSUM>
    ether 08:00:27:de:23:b6
    inet 192.168.6.2 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: em1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair51a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:60:4b:80:5e:0a
    inet6 fe80::60:4bff:fe80:5e0a%epair51a prefixlen 64 scopeid 0x8
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair52a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:be:7f:e3:53:0a
    inet6 fe80::be:7fff:fee3:530a%epair52a prefixlen 64 scopeid 0x9
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair61a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:77:ed:bd:4a:0a
    inet6 fe80::77:edff:febd:4a0a%epair61a prefixlen 64 scopeid 0xa
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair62a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:14:b4:95:10:0a
    inet6 fe80::14:b4ff:fe95:100a%epair62a prefixlen 64 scopeid 0xb
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


Destination        Gateway            Flags     Netif Expire
default            10.0.2.2           UGS         em0
10.0.2.0/24        link#1             U           em0
10.0.2.15          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.5.0/24     link#5             U         vlan5
192.168.5.2        link#5             UHS         lo0
192.168.6.0/24     link#7             U         vlan6
192.168.6.2        link#7             UHS         lo0
192.168.10.0/24    link#2             U           em1
192.168.10.101     link#2             UHS         lo0

jail51:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair51b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:60:4b:80:5e:0b
    inet6 fe80::60:4bff:fe80:5e0b%epair51b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:60:4b:80:5e:0b
    inet6 fe80::60:4bff:fe80:5e0b%vlan5 prefixlen 64 tentative scopeid 0x3
    inet 192.168.5.51 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: epair51b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.5.0/24     link#3             U         vlan5
192.168.5.51       link#3             UHS         lo0

jail52:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair52b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:be:7f:e3:53:0b
    inet6 fe80::be:7fff:fee3:530b%epair52b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:be:7f:e3:53:0b
    inet6 fe80::be:7fff:fee3:530b%vlan5 prefixlen 64 tentative scopeid 0x3
    inet 192.168.5.52 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: epair52b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.5.0/24     link#3             U         vlan5
192.168.5.52       link#3             UHS         lo0

jail61:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair61b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:77:ed:bd:4a:0b
    inet6 fe80::77:edff:febd:4a0b%epair61b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:77:ed:bd:4a:0b
    inet6 fe80::77:edff:febd:4a0b%vlan6 prefixlen 64 tentative scopeid 0x3
    inet 192.168.6.61 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: epair61b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.6.0/24     link#3             U         vlan6
192.168.6.61       link#3             UHS         lo0

jail62:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair62b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:14:b4:95:10:0b
    inet6 fe80::14:b4ff:fe95:100b%epair62b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:14:b4:95:10:0b
    inet6 fe80::14:b4ff:fe95:100b%vlan6 prefixlen 64 tentative scopeid 0x3
    inet 192.168.6.62 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: epair62b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.6.0/24     link#3             U         vlan6
192.168.6.62       link#3             UHS         lo0
 
Allright, because I do not intend to have a jail in multiple VLANs I was able to simplify things a bit by not using VLAN interfaces inside the jails. Between the jails and the host everything stays untagged, and jails on different VLANs are still isolated because they are on different bridges:

freebsd_vlan_jail_u-png.6762


Now I am also able to ping between the jails and the host.

After some tcpdump'ing it seems that ARP traffic does not get forwarded correctly through the bridges when trying to connect between a jail and the outside network. The jail sends a who-has request, which leaves the machine through the vlan5 or vlan6 interface, but a reply never comes back in. Does anyone hav any idea why ARP only seems to works correctly between jails or from host to outside network, but not from jails to outside network?
 

Attachments

  • freebsd_vlan_jail_u.png
    freebsd_vlan_jail_u.png
    138.4 KB · Views: 3,403
I have never done this before. But have you observed the ARP request leaving through the trunk? Have you been able to observe where the reply goes?

I could imagine that the issue could be that: you need to add em1 to your bridges, so that it forwards L2 traffic instead of just routing.

Edit: Or, maybe what I should have said is: create a third bridge with the two VLAN interfaces and em1 as members.
 
I had a similar problem. In my case, I was running the host on top of an ESXi. When I ran tcpdump, I saw "who-has" requests, but the replies where lost. It started working when I enabled "promiscuous mode" in the vswitch configuration in VMWare.
 
Your first picture isn't going to work. Traffic on em1 is tagged, traffic on bridge5 is untagged due to vlan5, so the VLAN info is already stripped off when the traffic is passed to your jails.
 
I can see the ARP requests leaving on the network because other devices (such as my UniFi gateway) receive them on their correct VLAN interfaces (so they get tagged correctly by FreeBSD and then get forwarded correctly by the switch) and send out replies on the same VLAN when I tcpdump at their end, the replies still do not come back in though. I have a UniFi switch, so I also tried to set another port to mirror the server port and sniff it to see if the ARP reply actually comes in, but nothing.

Looks like this user had a similar issue: Thread 68781
I also use the same driver with onboard NICs, I surely hope I do not need to spend extra on an expensive server NIC, this is already a X8SIL-F Supermicro server board...

pkc said:
Edit: Or, maybe what I should have said is: create a third bridge with the two VLAN interfaces and em1 as members.
Click to expand...

As far as I know vlan5 and vlan6 automatically receive all tagged packets that arrive on the physical interface and em1 gets all untagged traffic. I want to keep these separated. Wouldn't I lose this isolation if I bridge these interfaces? (e.g. packet comes in on vlan5, gets stripped, then gets tagged again for vlan6)

matiasb said:
I had a similar problem. In my case, I was running the host on top of an ESXi. When I ran tcpdump, I saw "who-has" requests, but the replies where lost. It started working when I enabled "promiscuous mode" in the vswitch configuration in VMWare.
Click to expand...

In this case I'm working on physical hardware (the MAC addresses in my starting post are of a VirtualBox test setup, where I got the same problem, but it is the same on my physical server board). When all interfaces are in promiscuous mode nothing changes unfortunately.

SirDice said:
Your first picture isn't going to work. Traffic on em1 is tagged, traffic on bridge5 is untagged due to vlan5, so the VLAN info is already stripped off when the traffic is passed to your jails.
Click to expand...

Yeah, I see where that goes wrong, that caused the connectivity problems between host and jails. The new situation should be fine though, where everything behind the bridges is untagged. Now I only have connectivity problems left between jails and the outside network. Host to network and host to jail are fine now.
 
Sfynx said:
Yeah, I see where that goes wrong, that caused the connectivity problems between host and jails. The new situation should be fine though, where everything behind the bridges is untagged. Now I only have connectivity problems left between jails and the outside network. Host to network and host to jail are fine now.
Click to expand...

I don't see any default gateways in your jails.
You need something like this:

default 192.168.6.1 UGS epair61b
 
Rudy said:
I don't see any default gateways in your jails.
You need something like this:

default 192.168.6.1 UGS epair61b
Click to expand...

Since my removal of the VLAN interfaces from the jails they do have a default gateway, situation is now:

Code: 
Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.6.1       UGS    epair61
127.0.0.1          link#1             UH          lo0
192.168.6.0/24    link#3             U      epair61
192.168.6.61       link#3             UHS         lo0

But I cannot reach the gateway itself from the jails anyway. Tagged ARP return traffic somehow gets dropped by the VLAN interfaces on the host, so it does not reach the jails through the bridges, so any hosts outside of the box never end up in the jail's ARP table, including the gateway itself. Only the host itself and the other jails on the same bridge ever get populated in the ARP table of a jail... which suggests that the untagged ARP traffic flows correctly through the bridges, it just gets lost when it comes from outside on a tagged interface.

Also tried to remove any VLAN related hardware flags (such as vlanhwfilter) from the physical interface in case those could be misbehaving, no difference. Tried a couple of USB Ethernet interfaces that required different drivers (Realtek, AX Media), no difference. Still hoping it is a configuration issue somewhere. Host talks with rest of network on tagged interface, so also cannot see how the switch would misbehave here, ports are set to allow all tagged VLANS (UniFi ALL profile) and when I disable tagged VLAN 5 or 6 on those ports the traffic stops.
 
Allright, it is working now and I feel a little stupid. I failed to mention I also have an untagged bridge0 with em1 itself as a member, assuming this would bridge only the untagged traffic coming in from em1, leaving the tagged traffic for the other bridges.
It turns out this bridge with the main interface as a member actually interferes with tagged traffic on other bridges, it seems it actually captures the traffic before the VLAN interfaces can act, causing things to get silently dropped. As soon as I removed this bridge all jails on the other bridges gained full connectivity with everything on the VLAN the bridges were attached to.

Lesson learned: do not assume too quickly that a piece of configuration is not relevant, and second: only use tagged interfaces in this case and do not use the parent interface at all.
 
Sfynx said:
Lesson learned: do not assume too quickly that a piece of configuration is not relevant, and second: only use tagged interfaces in this case and do not use the parent interface at all.
Click to expand...
And what if you do need VNET interface inside a jail to make it vlandev for other VLANs?
 
sol289 said:
And what if you do need VNET interface inside a jail to make it vlandev for other VLANs?
Click to expand...
Not sure, I haven't had a need for that yet. At this moment each jail is living in a single VLAN so it can just be attached to a single bridge that is part of that VLAN, and I let my UniFi gateway route between VLANs where ncessary.
 
You must log in or register to reply here.
Back
Top