Solved bhyve networking issues with ix, lagg, tap, and bridge

I am having trouble with setting up reliable networking for my bhyve host. My preferred configuration is:

FreeBSD 11.0 host and guests.
2 10gbe adapters on the host (ix0 & ix1)
combined into a link aggregation group (lagg0)
which is on a vlan trunk (lagg0.20)

So in this configuration, lagg0.20 provides access to vlan 20 on the physical network.

In order to provide network access to the guest OS, my configuration is to bridge a tap device with the lagg and to attach thes guests to tap20.

Issue #1

ifconfig bridge20 addm tap20 addm lagg0.20

creates the bridge, adds both interfaces, but also issues an error: “bridge20: error setting interface capabilities on lagg0.20”

Issue #2

Even with Issue #1, the bridge is created and the guest OS has access to the network. However, the performance on the guest is horrible. For example, downloading the latest copy of FreeBSD from ftp.freebsd.org gets about 20 kBps.

A tcpdump taken at different points reveals the following (notice the errors on the tap20 trace):

Code:
From Host on ix0 (cable unplugged from ix1 to force all traffic through 1 interfaces to improve tracing)

10:17:27.665269 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 117289, win 1026, options [nop,nop,TS val 43120869 ecr 1464866505], length 0

10:17:27.700523 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 117289:118737, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700584 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 118737:120185, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700844 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 981, options [nop,nop,TS val 43120904 ecr 1464866649], length 0

10:17:27.738879 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:123081, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 2896

10:17:27.739218 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 123081:124529, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739313 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 124529:125977, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739484 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:124529}], length 0

10:17:27.739490 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:125977}], length 0

10:17:27.774344 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464866723 ecr 43120943], length 2896

10:17:28.209608 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:121633, ack 1, win 32, options [nop,nop,TS val 1464867158 ecr 43120943], length 1448

10:17:28.209959 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 121633, win 1003, options [nop,nop,TS val 43121414 ecr 1464867158,nop,nop,sack 1 {123081:125977}], length 0

10:17:28.699958 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 121633:123081, ack 1, win 32, options [nop,nop,TS val 1464867649 ecr 43121414], length 1448

10:17:28.700319 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 125977, win 958, options [nop,nop,TS val 43121905 ecr 1464867649], length 0

10:17:28.739207 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464867687 ecr 43121905], length 2896

10:17:29.178086 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:127425, ack 1, win 32, options [nop,nop,TS val 1464868127 ecr 43121905], length 1448

10:17:29.285638 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 127425, win 1026, options [nop,nop,TS val 43122491 ecr 1464868127], length 0

10:17:29.324593 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:130321, ack 1, win 32, options [nop,nop,TS val 1464868274 ecr 43122491], length 2896

10:17:29.810250 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:128873, ack 1, win 32, options [nop,nop,TS val 1464868758 ecr 43122491], length 1448

10:17:29.922587 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 128873, win 1026, options [nop,nop,TS val 43123129 ecr 1464868758], length 0

Code:
From Host on lagg0.20

10:17:27.665265 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 117289, win 1026, options [nop,nop,TS val 43120869 ecr 1464866505], length 0

10:17:27.700526 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 117289:118737, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700585 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 118737:120185, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700839 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 981, options [nop,nop,TS val 43120904 ecr 1464866649], length 0

10:17:27.738885 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:123081, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 2896

10:17:27.739222 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 123081:124529, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739315 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 124529:125977, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739479 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:124529}], length 0

10:17:27.739489 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:125977}], length 0

10:17:27.774350 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464866723 ecr 43120943], length 2896

10:17:28.209613 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:121633, ack 1, win 32, options [nop,nop,TS val 1464867158 ecr 43120943], length 1448

10:17:28.209946 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 121633, win 1003, options [nop,nop,TS val 43121414 ecr 1464867158,nop,nop,sack 1 {123081:125977}], length 0

10:17:28.699963 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 121633:123081, ack 1, win 32, options [nop,nop,TS val 1464867649 ecr 43121414], length 1448

10:17:28.700315 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 125977, win 958, options [nop,nop,TS val 43121905 ecr 1464867649], length 0

10:17:28.739214 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464867687 ecr 43121905], length 2896

10:17:29.178090 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:127425, ack 1, win 32, options [nop,nop,TS val 1464868127 ecr 43121905], length 1448

10:17:29.285635 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 127425, win 1026, options [nop,nop,TS val 43122491 ecr 1464868127], length 0

10:17:29.324598 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:130321, ack 1, win 32, options [nop,nop,TS val 1464868274 ecr 43122491], length 2896

10:17:29.810255 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:128873, ack 1, win 32, options [nop,nop,TS val 1464868758 ecr 43122491], length 1448

10:17:29.922583 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 128873, win 1026, options [nop,nop,TS val 43123129 ecr 1464868758], length 0

Code:
From Host on tap20

10:17:27.665261 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 117289, win 1026, options [nop,nop,TS val 43120869 ecr 1464866505], length 0

10:17:27.700595 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 117289:118737, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700602 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 118737:120185, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

10:17:27.700835 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 981, options [nop,nop,TS val 43120904 ecr 1464866649], length 0

10:17:27.738960 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:123081, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 2896

10:17:27.739303 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 123081:124529, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739325 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 124529:125977, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

10:17:27.739476 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:124529}], length 0

10:17:27.739488 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:125977}], length 0

10:17:27.774422 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464866723 ecr 43120943], length 2896

10:17:28.209685 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:121633, ack 1, win 32, options [nop,nop,TS val 1464867158 ecr 43120943], length 1448

10:17:28.209941 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 121633, win 1003, options [nop,nop,TS val 43121414 ecr 1464867158,nop,nop,sack 1 {123081:125977}], length 0

10:17:28.700038 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 121633:123081, ack 1, win 32, options [nop,nop,TS val 1464867649 ecr 43121414], length 1448

10:17:28.700312 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 125977, win 958, options [nop,nop,TS val 43121905 ecr 1464867649], length 0

10:17:28.739283 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464867687 ecr 43121905], length 2896

10:17:29.178160 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:127425, ack 1, win 32, options [nop,nop,TS val 1464868127 ecr 43121905], length 1448

10:17:29.285632 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 127425, win 1026, options [nop,nop,TS val 43122491 ecr 1464868127], length 0

10:17:29.324670 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:130321, ack 1, win 32, options [nop,nop,TS val 1464868274 ecr 43122491], length 2896

10:17:29.810326 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:128873, ack 1, win 32, options [nop,nop,TS val 1464868758 ecr 43122491], length 1448

10:17:29.922579 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 128873, win 1026, options [nop,nop,TS val 43123129 ecr 1464868758], length 0

Code:
From Guest on vtnet0

05:18:17.280509 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 117289, win 1026, options [nop,nop,TS val 43120869 ecr 1464866505], length 0

05:18:17.316083 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 117289:118737, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

05:18:17.316107 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 118737:120185, ack 1, win 32, options [nop,nop,TS val 1464866649 ecr 43120869], length 1448

05:18:17.316121 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 981, options [nop,nop,TS val 43120904 ecr 1464866649], length 0

05:18:17.354474 IP truncated-ip - 926 bytes missing! 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:123081, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 2896

05:18:17.354805 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 123081:124529, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

05:18:17.354824 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:124529}], length 0

05:18:17.354849 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 124529:125977, ack 1, win 32, options [nop,nop,TS val 1464866688 ecr 43120904], length 1448

05:18:17.354858 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 120185, win 1026, options [nop,nop,TS val 43120943 ecr 1464866649,nop,nop,sack 1 {123081:125977}], length 0

05:18:17.389995 IP truncated-ip - 926 bytes missing! 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464866723 ecr 43120943], length 2896

05:18:17.825773 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 120185:121633, ack 1, win 32, options [nop,nop,TS val 1464867158 ecr 43120943], length 1448

05:18:17.825817 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 121633, win 1003, options [nop,nop,TS val 43121414 ecr 1464867158,nop,nop,sack 1 {123081:125977}], length 0

05:18:18.316695 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 121633:123081, ack 1, win 32, options [nop,nop,TS val 1464867649 ecr 43121414], length 1448

05:18:18.316762 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 125977, win 958, options [nop,nop,TS val 43121905 ecr 1464867649], length 0

05:18:18.355986 IP truncated-ip - 926 bytes missing! 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:128873, ack 1, win 32, options [nop,nop,TS val 1464867687 ecr 43121905], length 2896

05:18:18.795342 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 125977:127425, ack 1, win 32, options [nop,nop,TS val 1464868127 ecr 43121905], length 1448

05:18:18.902763 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 127425, win 1026, options [nop,nop,TS val 43122491 ecr 1464868127], length 0

05:18:18.942051 IP truncated-ip - 926 bytes missing! 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:130321, ack 1, win 32, options [nop,nop,TS val 1464868274 ecr 43122491], length 2896

05:18:19.428286 IP 96.47.72.72.51315 > 192.168.1.93.29996: Flags [.], seq 127425:128873, ack 1, win 32, options [nop,nop,TS val 1464868758 ecr 43122491], length 1448

05:18:19.540434 IP 192.168.1.93.29996 > 96.47.72.72.51315: Flags [.], ack 128873, win 1026, options [nop,nop,TS val 43123129 ecr 1464868758], length 0


It seems as if the network card is combining some packets before delivering them to the OS (which I understand is a good and normal thing). You can see that the payload of some packets exceeds the MTU of the interface. This does not cause any trouble on the host (network performance for the same tasks is easily 3000 kBps). The packets travel successfully through the stack (ix0 -> lagg0.20 -> tap20) but the interface on the guest (vtnet0) does not like the larger packets. This causes many retransmissions and poor performance.

Jacking up the mtu on the guest interface to 16384 (nice round number) improves performance but tcpdump still shows occasional issues:

Code:
05:38:32.861350 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1960592:1962040, ack 1, win 32, options [nop,nop,TS val 1064751443 ecr 44336415], length 1448

05:38:32.861379 IP 192.168.1.93.45247 > 96.47.72.72.55159: Flags [.], ack 1962040, win 1771, options [nop,nop,TS val 44336450 ecr 1064751419], length 0

05:38:32.861400 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1962040:1963488, ack 1, win 32, options [nop,nop,TS val 1064751443 ecr 44336415], length 1448

05:38:32.861954 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1963488:1964936, ack 1, win 32, options [nop,nop,TS val 1064751443 ecr 44336415], length 1448

05:38:32.861979 IP 192.168.1.93.45247 > 96.47.72.72.55159: Flags [.], ack 1964936, win 1771, options [nop,nop,TS val 44336450 ecr 1064751443], length 0

05:38:32.861998 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1964936:1967832, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 2896

05:38:32.862008 IP 192.168.1.93.45247 > 96.47.72.72.55159: Flags [.], ack 1967832, win 1726, options [nop,nop,TS val 44336450 ecr 1064751444], length 0

05:38:32.862013 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1967832:1969280, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 1448

05:38:32.862752 IP truncated-ip - 326 bytes missing! 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1969280:1973624, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 4344

05:38:32.862761 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1973624:1975072, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 1448

05:38:32.862777 IP 192.168.1.93.45247 > 96.47.72.72.55159: Flags [.], ack 1969280, win 1794, options [nop,nop,TS val 44336451 ecr 1064751444,nop,nop,sack 1 {1973624:1975072}], length 0

05:38:32.863326 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1975072:1976520, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 1448

05:38:32.863352 IP 192.168.1.93.45247 > 96.47.72.72.55159: Flags [.], ack 1969280, win 1794, options [nop,nop,TS val 44336451 ecr 1064751444,nop,nop,sack 1 {1973624:1976520}], length 0

05:38:32.863383 IP 96.47.72.72.55159 > 192.168.1.93.45247: Flags [.], seq 1976520:1977968, ack 1, win 32, options [nop,nop,TS val 1064751444 ecr 44336415], length 1448

The double sized packets make it through, but triple sized packets still generate errors.


Can anyone help?
 
Last edited by a moderator:
#1 I believe you get this error if the MTU sizes of the interfaces is different. Make sure all MTU settings are the same. Bridging two interfaces with different MTU sizes is going to cause problems.
 
Turns on the problem with issue #1 is not mtu related, but a problem with virtual devices in a bridge. I noticed the following:

Here is a physical interface before being part of a bridge:

Code:
ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:d3:0d:cf
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier

However, after being added to a bridge, the options change:

Code:
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:d3:0d:cf
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier

Adding the physical interface to the bridge removes the following options:

TXCSUM, TSO6, TSO4, and LRO

I think that when you add a virtual interface to a bridge, the bridge tries to disable those options on the virtual interface and fails which generates the message "error setting interface capabilities on ..."

If I manually disable those options on the underlying physical interfaces for the lagg, the host stops getting the "oversized" packets and the guest no longer has any trouble.

I hope this helps someone else in the future.

Any advice on a better place to post this theory so that it can be addressed in a future release?
 
Back
Top