Solved FreeBSD bridge not recieving ARP responses

K

keleathi

I'm attempting to get a vnet jail to communicate with the LAN via a bridge to the primary physical interface, but I'm unable to get it to work.
I'm running FreeBSD 12
uname -a FreeBSD hostname 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC amd64

I found Thread 19708 which seems to describe my exact problem, but had no resolution I could figure out.

I have no firewalls or filtering on this system.

My ifconfig output is as follows:
Code: 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 40:8d:5c:c4:cc:55
        inet 192.168.6.30 netmask 0xffffff00 broadcast 192.168.6.255
        inet 192.168.6.52 netmask 0xffffffff broadcast 192.168.6.52
        inet 192.168.6.53 netmask 0xffffffff broadcast 192.168.6.53
        inet 192.168.6.47 netmask 0xffffffff broadcast 192.168.6.47
        inet 192.168.6.44 netmask 0xffffffff broadcast 192.168.6.44
        inet 192.168.6.42 netmask 0xffffffff broadcast 192.168.6.42
        inet 192.168.6.51 netmask 0xffffffff broadcast 192.168.6.51
        inet 192.168.6.45 netmask 0xffffffff broadcast 192.168.6.45
        inet 192.168.6.49 netmask 0xffffffff broadcast 192.168.6.49
        inet 192.168.6.43 netmask 0xffffffff broadcast 192.168.6.43
        inet 192.168.6.48 netmask 0xffffffff broadcast 192.168.6.48
        inet 192.168.6.46 netmask 0xffffffff broadcast 192.168.6.46
        inet 192.168.6.31 netmask 0xffffffff broadcast 192.168.6.31
        inet 192.168.6.36 netmask 0xffffffff broadcast 192.168.6.36
        inet 192.168.6.35 netmask 0xffffffff broadcast 192.168.6.35
        inet 192.168.6.33 netmask 0xffffffff broadcast 192.168.6.33
        inet 192.168.6.38 netmask 0xffffffff broadcast 192.168.6.38
        inet 192.168.6.39 netmask 0xffffffff broadcast 192.168.6.39
        inet 192.168.6.34 netmask 0xffffffff broadcast 192.168.6.34
        inet 192.168.6.37 netmask 0xffffffff broadcast 192.168.6.37
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.1.52 netmask 0xffffffff
        inet 127.0.1.53 netmask 0xffffffff
        inet 127.0.1.51 netmask 0xffffffff
        inet 127.0.1.1 netmask 0xffffffff
        inet 127.0.1.6 netmask 0xffffffff
        inet 127.0.1.5 netmask 0xffffffff
        inet 127.0.1.3 netmask 0xffffffff
        inet 127.0.1.8 netmask 0xffffffff
        inet 127.0.1.9 netmask 0xffffffff
        inet 127.0.1.4 netmask 0xffffffff
        inet 127.0.1.7 netmask 0xffffffff
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:66:3a:87:e6:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
ue0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether d8:eb:97:bd:2d:dd
        inet 10.43.0.53 netmask 0xffffffff broadcast 10.43.0.53
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: poudriere_20190427 as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:89:12:77
        hwaddr 02:df:26:83:a8:0a
        inet6 fe80::ff:60ff:fe89:1277%vnet0.6 prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

My sysctls regarding the bridge are:
Code: 
# sysctl -a |grep net.link.bridge

net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0

Currently, port 80 is listening in the jail (with IP of 192.168.6.50).

Running nc 192.168.6.50 80 from an external machine gets me the following packet captures:

Code: 
# tcpdump -eni re0 host 192.168.6.50

14:49:32.812738 00:0d:b9:34:d5:51 > 02:ff:60:89:12:78, ethertype IPv4 (0x0800), length 74: 192.168.6.1.10808 > 192.168.6.50.80: Flags [S], seq 4259154533, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1118660762 ecr 0], length 0
14:49:32.812823 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:32.813016 00:0d:b9:34:d5:51 > 02:ff:60:89:12:78, ethertype ARP (0x0806), length 60: Reply 192.168.6.1 is-at 00:0d:b9:34:d5:51, length 46
14:49:35.812570 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:35.812833 00:0d:b9:34:d5:51 > 02:ff:60:89:12:78, ethertype ARP (0x0806), length 60: Reply 192.168.6.1 is-at 00:0d:b9:34:d5:51, length 46
14:49:39.014516 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:39.014826 00:0d:b9:34:d5:51 > 02:ff:60:89:12:78, ethertype ARP (0x0806), length 60: Reply 192.168.6.1 is-at 00:0d:b9:34:d5:51, length 46
14:49:42.214476 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:42.214638 00:0d:b9:34:d5:51 > 02:ff:60:89:12:78, ethertype ARP (0x0806), length 60: Reply 192.168.6.1 is-at 00:0d:b9:34:d5:51, length 46
Code: 
# tcpdump -eni vnet0.6 host 192.168.6.50

14:49:32.812777 40:8d:5c:c4:cc:55 > 02:ff:60:89:12:78, ethertype IPv4 (0x0800), length 74: 192.168.6.1.10808 > 192.168.6.50.80: Flags [S], seq 4259154533, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1118660762 ecr 0], length 0
14:49:32.812821 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:35.812558 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:39.014503 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28
14:49:42.214470 02:ff:60:89:12:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.1 tell 192.168.6.50, length 28

As you can see, the SYN packet makes it in through the bridge, ARP requests make their way out of the bridge, but ARP replies don't.

Thank you in advance!
 
New data point: after allowing the ARP cache to expire on the external machine, the external ARP requests aren't getting in either. Again, running nc 192.168.6.50 80 from the external machine:

Code: 
# tcpdump -eni re0 host 192.168.6.50

15:15:15.730849 00:0d:b9:34:d5:51 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.50 tell 192.168.6.1, length 46
15:15:18.730882 00:0d:b9:34:d5:51 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.50 tell 192.168.6.1, length 46

Code: 
# tcpdump -eni vnet0.6 host 192.168.6.50

<nothing>

so it seems as if ARP in general from the external LAN is just not making it through the bridge.
 
This is now fixed (or at least worked around).
For posterity: it seems to be some sort of defect or constraint between virtualbox (version 5.2.22) and if_bridge. Stopping all of the virtualbox VMs allows this to work without issue.
 
You must log in or register to reply here.
Back
Top