Solved Partial connectivity issues from bhyve guests to jails and host*

Please help, I have been trying to figure this out for a couple of weeks now. I need a new set of eyes on this problem. Attached is a diagram to better illustrate the configuration.

To sum up the issue:
  • Can ping any host to any host
  • Can fully communicate from/to other physical hosts to the FreeBSD host
  • Can fully communicate from/to other physical hosts to the jails
  • Can fully communicate from/to other physical hosts to the bhyve guests
  • Can fully communicate from bhyve3 (PCI passthrough) to any other device
  • Can not communicate beyond pings to/from bhyve 1&2 (taps) to/from the jails
  • Can not communicate beyond pings to/from bhyve 1&2 (taps) to/from the host
  • Can fully communicate between bhyve1 and bhyve2
I am defining "fully communicate" as ssh, dns, http, and/or etc.

/etc/rc.conf excerpt
Code:
defaultrouter="10.0.70.22"
cloned_interfaces="lagg0 bridge70 \
                   tap70 tap7000 tap7001"
ifconfig_bce0="up"
ifconfig_bce1="up"

ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1"
vlans_lagg0="v070 v075"
create_args_v070="vlan 70"
ifconfig_v070="inet 10.0.70.30 netmask 255.255.255.0 up"
ifconfig_bridge70="up addm v070 addm tap70 addm tap7000 addm tap7001"

root@10.0.70.30: ~# ifconfig
Code:
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: bce1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge70: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
        ether 02:02:4b:05:07:46
        nd6 options=9<PERFORMNUD,IFDISABLED>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap7001 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: tap7000 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: tap70 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000
        member: v070 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 31 priority 128 path cost 55
tap70: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:4f:b2:00:59
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        Opened by PID 41775
tap7000: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:59:b2:00:5a
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        Opened by PID 6221
tap7001: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:3a:b2:00:46
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
v070: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=103<RXCSUM,TXCSUM,TSO4>
        ether f0:4d:a2:07:d8:15
        inet 10.0.70.30 netmask 0xffffff00 broadcast 10.0.70.255
        inet 10.0.70.20 netmask 0xffffff00 broadcast 10.0.70.255
        inet 10.0.70.21 netmask 0xffffff00 broadcast 10.0.70.255
        inet 10.0.70.22 netmask 0xffffff00 broadcast 10.0.70.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 70 parent interface: lagg0
Code:
root@freebsd103host: ~# iocage get ip4_addr jail01
v070|10.0.70.20/24
root@freebsd103host: ~# iocage get ip4_addr jail02
v070|10.0.70.21/24
root@freebsd103host: ~# iocage get ip4_addr jail03
v070|10.0.70.22/24

Am I missing something in the configurations? Is this expected behavior?

I am using iohyve and iocage to manage the bhyve guest and jails respectively. I tried attaching the jail IPs to tap7001 to no avail. I also attempted to change the path cost values to no avail. I am not sure what else to try. Any suggestions would be appreciated.
 

Attachments

  • freebsd_tapbridgevlan_issue.png
    freebsd_tapbridgevlan_issue.png
    119.8 KB · Views: 400
I found the solution!

I started running tcpdump with the -vvv flag and noticed checksum errors. Remembering back when I ran pfSense on bare metal, I remember turning off TSO and anything to do with hardware offloading in the advanced options. It does not quite make sense to me considering the packets having the issues never touched the hardware interfaces but it worked. If anyone has an explanation I would love to hear it.

The fix was removing all the hardware offload capabilities for the Broadcom NICs, turning off TSO globally, then rebooting.

/etc/rc.conf
Code:
ifconfig_bce0="-rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwfilter -vlanhwtso -tso -tso4 -tso6 -lro -vlanhwtso -vlanhwcsum up"
ifconfig_bce1="-rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtag -vlanhwfilter -vlanhwtso -tso -tso4 -tso6 -lro -vlanhwtso -vlanhwcsum up"

/boot/loader.conf
Code:
hw.bce.tso_enable=0
net.inet.tcp.tso=0


Option for bce0 bce1 and lagg0 afterwards:
Code:
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether f0:4d:a2:07:d8:15
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: bce1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>



Checksum errors:
Code:
[NOPARSE]root@10.0.70.30:~ # tcpdump -vvv -i tap7001 | grep 10.0.70.20                                                                                                                                                                     
tcpdump: WARNING: tap7001: no IPv4 address assigned
tcpdump: listening on tap7001, link-type EN10MB (Ethernet), capture size 65535 bytes
20:59:30.715408 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.70.20 tell 10.0.70.11, length 46
20:59:30.715463 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.0.70.20 is-at f0:4d:a2:07:d8:15 (oui Unknown), length 28
    10.0.70.11.5149 > 10.0.70.20.domain: [udp sum ok] 57677+ A? google.com. (28)
20:59:30.716161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.70.20 tell 10.0.70.31, length 46
    10.0.70.20.domain > 10.0.70.11.5149: [bad udp cksum 0xa0dd -> 0x56fe!] 57677 q: A? google.com. 1/4/4 google.com. [5m] A 216.58.217.238 ns: google.com. [1d9h25m45s] NS ns1.google.com., google.com. [1d9h25m45s] NS ns4.google.com., google.com. [1d9h25m45s] NS ns3.google.com., google.com. [1d9h25m45s] NS ns2.google.com. ar: ns1.google.com. [1d1h21m46s] A 216.239.32.10, ns2.google.com. [1d9h23m24s] A 216.239.34.10, ns3.google.com. [1d9h23m24s] A 216.239.36.10, ns4.google.com. [1d9h23m24s] A 216.239.38.10 (180)
21:01:02.321663 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.70.20 tell 10.0.70.3, length 46
    10.0.70.11.39029 > 10.0.70.20.http: Flags [S], cksum 0x9232 (correct), seq 3365459431, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 51567768 ecr 0], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x585b), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51567768], length 0
    10.0.70.11.39029 > 10.0.70.20.http: Flags [S], cksum 0x867a (correct), seq 3365459431, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 51570768 ecr 0], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x4ca3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51570768], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x4ca3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51570768], length 0
    10.0.70.11.39029 > 10.0.70.20.http: Flags [S], cksum 0x79fa (correct), seq 3365459431, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 51573968 ecr 0], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x4023), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51573968], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x4023), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51573968], length 0
    10.0.70.11.39029 > 10.0.70.20.http: Flags [S], cksum 0x6d7a (correct), seq 3365459431, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 51577168 ecr 0], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x33a3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51577168], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x33a3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51577168], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x33a3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51577168], length 0
    10.0.70.20.http > 10.0.70.11.39029: Flags [S.], cksum 0xa03e (incorrect -> 0x33a3), seq 332226024, ack 3365459432, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2585537730 ecr 51577168], length 0[/NOPARSE]

I hope this helps someone else from hours of troubleshooting. Had I been a little more attentive reading the pfSense documentention I might of caught this:
Several users have noted issues with certain Broadcom network cards, especially those built into Dell hardware. If the bce cards in the firewall are behaving erratically, dropping packets, or causing system crashes, then the following tweaks may help, especially on amd64.
 
Back
Top