Bridge/epair not passing through tagged VLAN traffic between host and VNET jail

Sfynx

Active Member

Reaction score: 13
Messages: 116

I have worked with FreeBSD and/or VLANs for quite a while now, but now I have something I cannot figure out. Hope someone can tell me what I am doing wrong.

I want to set up a FreeBSD box with VNET enabled jails where I can put each jail into a specific VLAN, all accessible via a single physical interface connected to a VLAN aware switch. For this, I have created a bridgeX interface for each VLAN. On each bridge I then put a single VLAN host interface and one or more epair interfaces, in other words, quite the standard recipe for VNET based jails. By the way, I was inspired by this guide: https://gist.github.com/sdebnath/086874c5df8b68e0df69

I guess one image says more than a thousand words in this case, this is my setup:

6761


Now onto why I wrote this forum post: it seems I cannot connect between the host and the jails this way. I cannot ping 192.168.5.51 from the host, and I cannot ping to the host (192.168.5.2) from either the 192.168.5.51 or 192.168.5.52 jails. Traffic from outside the box will also not reach the jails and vice versa.

What does work:
  • Pinging the 192.168.5.2 and 192.168.6.2 host IP addresses from outside the box when being in the respective VLANs. This indicates that the switch trunk port is configured correctly and the VLAN interfaces on the host are accepting the tagged packets.
  • Pinging between jails connected to the same bridge, e.g. pinging 192.168.5.52 from 192.168.5.51. So the bridges and epairs are forwarding packets as long as they do not go to the host.
What does not work:
  • Pinging from the host to a jail, or from a jail to the host. For instance, I cannot send anything between 192.168.5.2 and 192.168.5.51 or the other way around.
  • Pinging a jail IP from outside the box when connected to the correct VLAN. I can only ping the host itself.
In other words: it seems that something (bridge? epair?) is blocking communication between host and jails, allowing only traffic from the network to the host, or traffic between jails. I do not have any firewall such as pf enabled yet, I'd rather get this working before making it more complex by filtering any packets. Enabling IP forwarding does not fix anything and IMO would not be needed for this anyway, as in my view this should still be layer 2 material.

I have included the ifconfig and routing table information of the host and all jail environments below, I do not see anything unexpected here, but maybe you can. So there it is, can someone point me to the obvious thing I forgot so I can go on and slam my head into my desk? ;)

Used version: 12.0-RELEASE-p8

Host:
Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
    ether 08:00:27:6d:e1:73
    inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
    ether 08:00:27:de:23:b6
    inet 192.168.10.101 netmask 0xffffff00 broadcast 192.168.10.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e3:55:df:8c:05
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair52a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: epair51a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: vlan5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=1<RXCSUM>
    ether 08:00:27:de:23:b6
    inet 192.168.5.2 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: em1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e3:55:df:8c:06
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair62a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: epair61a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: vlan6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=1<RXCSUM>
    ether 08:00:27:de:23:b6
    inet 192.168.6.2 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: em1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair51a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:60:4b:80:5e:0a
    inet6 fe80::60:4bff:fe80:5e0a%epair51a prefixlen 64 scopeid 0x8
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair52a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:be:7f:e3:53:0a
    inet6 fe80::be:7fff:fee3:530a%epair52a prefixlen 64 scopeid 0x9
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair61a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:77:ed:bd:4a:0a
    inet6 fe80::77:edff:febd:4a0a%epair61a prefixlen 64 scopeid 0xa
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair62a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:14:b4:95:10:0a
    inet6 fe80::14:b4ff:fe95:100a%epair62a prefixlen 64 scopeid 0xb
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


Destination        Gateway            Flags     Netif Expire
default            10.0.2.2           UGS         em0
10.0.2.0/24        link#1             U           em0
10.0.2.15          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.5.0/24     link#5             U         vlan5
192.168.5.2        link#5             UHS         lo0
192.168.6.0/24     link#7             U         vlan6
192.168.6.2        link#7             UHS         lo0
192.168.10.0/24    link#2             U           em1
192.168.10.101     link#2             UHS         lo0
jail51:
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair51b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:60:4b:80:5e:0b
    inet6 fe80::60:4bff:fe80:5e0b%epair51b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:60:4b:80:5e:0b
    inet6 fe80::60:4bff:fe80:5e0b%vlan5 prefixlen 64 tentative scopeid 0x3
    inet 192.168.5.51 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: epair51b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.5.0/24     link#3             U         vlan5
192.168.5.51       link#3             UHS         lo0
jail52:
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair52b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:be:7f:e3:53:0b
    inet6 fe80::be:7fff:fee3:530b%epair52b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:be:7f:e3:53:0b
    inet6 fe80::be:7fff:fee3:530b%vlan5 prefixlen 64 tentative scopeid 0x3
    inet 192.168.5.52 netmask 0xffffff00 broadcast 192.168.5.255
    groups: vlan
    vlan: 5 vlanpcp: 0 parent interface: epair52b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.5.0/24     link#3             U         vlan5
192.168.5.52       link#3             UHS         lo0
jail61:
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair61b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:77:ed:bd:4a:0b
    inet6 fe80::77:edff:febd:4a0b%epair61b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:77:ed:bd:4a:0b
    inet6 fe80::77:edff:febd:4a0b%vlan6 prefixlen 64 tentative scopeid 0x3
    inet 192.168.6.61 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: epair61b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.6.0/24     link#3             U         vlan6
192.168.6.61       link#3             UHS         lo0
jail62:
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair62b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:14:b4:95:10:0b
    inet6 fe80::14:b4ff:fe95:100b%epair62b prefixlen 64 tentative scopeid 0x2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:14:b4:95:10:0b
    inet6 fe80::14:b4ff:fe95:100b%vlan6 prefixlen 64 tentative scopeid 0x3
    inet 192.168.6.62 netmask 0xffffff00 broadcast 192.168.6.255
    groups: vlan
    vlan: 6 vlanpcp: 0 parent interface: epair62b
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#1             UH          lo0
192.168.6.0/24     link#3             U         vlan6
192.168.6.62       link#3             UHS         lo0
 
OP
OP
Sfynx

Sfynx

Active Member

Reaction score: 13
Messages: 116

Allright, because I do not intend to have a jail in multiple VLANs I was able to simplify things a bit by not using VLAN interfaces inside the jails. Between the jails and the host everything stays untagged, and jails on different VLANs are still isolated because they are on different bridges:



Now I am also able to ping between the jails and the host.

After some tcpdump'ing it seems that ARP traffic does not get forwarded correctly through the bridges when trying to connect between a jail and the outside network. The jail sends a who-has request, which leaves the machine through the vlan5 or vlan6 interface, but a reply never comes back in. Does anyone hav any idea why ARP only seems to works correctly between jails or from host to outside network, but not from jails to outside network?
 

Attachments

pkc

Member

Reaction score: 2
Messages: 79

I have never done this before. But have you observed the ARP request leaving through the trunk? Have you been able to observe where the reply goes?

I could imagine that the issue could be that: you need to add em1 to your bridges, so that it forwards L2 traffic instead of just routing.

Edit: Or, maybe what I should have said is: create a third bridge with the two VLAN interfaces and em1 as members.
 

matiasb

New Member


Messages: 1

I had a similar problem. In my case, I was running the host on top of an ESXi. When I ran tcpdump, I saw "who-has" requests, but the replies where lost. It started working when I enabled "promiscuous mode" in the vswitch configuration in VMWare.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,672
Messages: 30,574

Your first picture isn't going to work. Traffic on em1 is tagged, traffic on bridge5 is untagged due to vlan5, so the VLAN info is already stripped off when the traffic is passed to your jails.
 
Top