Update: I wanted to mention I solved this issue by implementing some (very few) of the CALOMEL tuning parameters and enabling tso + lro using ifconfig and sysctl. I will expand more later, but as of now Linux and FreeBSD throughput in ESXi appears to be essentially identical.
As re: CALOMEL recommendations + Netflix RACK kernel mod, they were interesting, but for my setup managed to harm my throughput more than help. My conclusion came after 5-6 hours of iperf3 testing host-host and loopback. I ended up sticking mostly with freebsd defaults. The tso/lro appeared to make the most noticable (several Gbps worth) of difference.
CALOMEL: https://calomel.org/freebsd_network_tuning.html
Post #6 of this thread gave me a great starting point to begin to nail down the issue: https://www.truenas.com/community/t...-is-poor-with-iperf3-tests.63173/#post-452479
(note, I am NOT using FreeNAS, but was similar enough scenario to be helpful)
Orig post:
I have a 3-server homelab using ESXi 7.0U1 and Intel 82599 10GbE NICs. I was benchmarking
Here's the FreeBSD 12.2 VM's info - I'm using
Here's the Ubuntu 20.04 VM I compared
Here's the VM I used for testing (on another host):
Here's the FreeBSD 12.2-RELEASE
Here's the Ubuntu 20.04 VM's
OPNsense, which is based on FreeBSD, also has poor 10GbE performance. But tests show versions of OPNSense based on FreeBSD 11 had better 10GbE throughput than FreeBSD 12.
Is this throughput discrepancy a regression in FreeBSD? Can it be fixed by end user, or is it something that must be accomplished upstream?
Sources:
forum.opnsense.org
forum.opnsense.org
Update: Some other interesting observations ---
FreeBSD
Ubuntu 20.04
FreeBSD to Ubuntu 20.04
Ubuntu 20.04 to FreeBSD
Wow...
As re: CALOMEL recommendations + Netflix RACK kernel mod, they were interesting, but for my setup managed to harm my throughput more than help. My conclusion came after 5-6 hours of iperf3 testing host-host and loopback. I ended up sticking mostly with freebsd defaults. The tso/lro appeared to make the most noticable (several Gbps worth) of difference.
CALOMEL: https://calomel.org/freebsd_network_tuning.html
Post #6 of this thread gave me a great starting point to begin to nail down the issue: https://www.truenas.com/community/t...-is-poor-with-iperf3-tests.63173/#post-452479
(note, I am NOT using FreeNAS, but was similar enough scenario to be helpful)
Orig post:
I have a 3-server homelab using ESXi 7.0U1 and Intel 82599 10GbE NICs. I was benchmarking
iperf3 scores on a few VMs and wanted to test against my FreeBSD VM, and I noticed its throughput is quite a few Gbps lower than other VMs I have in my lab.Here's the FreeBSD 12.2 VM's info - I'm using
inxi since it's cross-platform:
Code:
# uname -r
12.2-RELEASE-p1
# pciconf -lv | grep -C2 VMXNET
vmx0@pci0:11:0:0: class=0x020000 card=0x07b015ad chip=0x07b015ad rev=0x01 hdr=0x00
vendor = 'VMware'
device = 'VMXNET3 Ethernet Controller'
class = network
subclass = ethernet
# inxi -Nxxx
Network: Device-1: Intel 82371AB/EB/MB PIIX4 ACPI type: network bridge driver: intsmb port: N/A bus ID: 0:0:7.3
chip ID: 8086:7113
Device-2: VMware VMXNET3 Ethernet driver: vmx port: N/A bus ID: 0:11:0.0 chip ID: 15ad:07b0
#ifconfig -a
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6
>
ether 00:50:56:bf:1a:90
inet 192.168.1.71 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
# inxi -Cxxx
CPU: Info: Quad Core model: Intel Xeon E5-2630L v3 bits: 64 type: MCP arch: N/A L1 cache: 64 KiB L2 cache: N/A
features: avx avx2 lm nx pae sse sse2 sse3 ssse3
Speed: 1800 MHz min/max: N/A volts: 3.3 V Core speeds (MHz): No speed data found for 4 cores.Here's the Ubuntu 20.04 VM I compared
iperf3 test scores against:
Code:
# uname -a
Linux hostname 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# lspci -vnn | grep -A 14 VMXNET
0b:00.0 Ethernet controller [0200]: VMware VMXNET3 Ethernet Controller [15ad:07b0] (rev 01)
DeviceName: Ethernet0
Subsystem: VMware VMXNET3 Ethernet Controller [15ad:07b0]
Physical Slot: 192
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at fe213000 (32-bit, non-prefetchable) [size=4K]
Memory at fe212000 (32-bit, non-prefetchable) [size=4K]
Memory at fe210000 (32-bit, non-prefetchable) [size=8K]
I/O ports at 3000 [size=16]
Expansion ROM at fe200000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [48] Express Endpoint, MSI 00
Capabilities: [84] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [9c] MSI-X: Enable+ Count=65 Masked-
Capabilities: [100] Device Serial Number 00-50-56-ff-ff-9d-6b-02
Kernel driver in use: vmxnet3
Kernel modules: vmxnet3
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:9d:6b:02 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.28/24 brd 192.168.1.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 2601:603:4d00:508::111f/128 scope global dynamic noprefixroute
valid_lft 6996sec preferred_lft 2715sec
inet6 2601:603:4d00:508:250:56ff:fe9d:6b02/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86324sec preferred_lft 14324sec
inet6 fe80::250:56ff:fe9d:6b02/64 scope link
valid_lft forever preferred_lft forever
# inxi -Nxxx
Network: Device-1: Intel 82371AB/EB/MB PIIX4 ACPI vendor: VMware Virtual Machine type: network bridge driver: N/A
port: 2050 bus ID: 00:07.3 chip ID: 8086:7113
Device-2: VMware VMXNET3 Ethernet driver: vmxnet3 v: 1.4.17.0-k port: 3000 bus ID: 0b:00.0 chip ID: 15ad:07b0
# inxi -Cxxx
CPU: Topology: 4x Single Core model: Intel Xeon E5-2630L v3 bits: 64 type: SMP arch: Haswell rev: 2
L2 cache: 80.0 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 bogomips: 14399
Speed: 1800 MHz min/max: N/A Core speeds (MHz): 1: 1800 2: 1800 3: 1800 4: 1800Here's the VM I used for testing (on another host):
Code:
# uname -a
SunOS hostname 5.11 omnios-r151030-6faec2dd4d i86pc i386 i86pc illumos
# modinfo -w -i 198
Id Loadaddr Size Info Rev Module Name
198 fffffffff7e10000 4db8 247 1 vmxnet3s (VMware EtherAdapter v3 1.1.0.0)
# lspci -nnv | grep -A 10 VMXNET
03:00.0 Ethernet controller [0200]: VMware VMXNET3 Ethernet Controller [15ad:07b0] (rev 01)
Subsystem: VMware VMXNET3 Ethernet Controller [15ad:07b0]
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at fd5fc000 (32-bit, non-prefetchable)
Memory at fd5fd000 (32-bit, non-prefetchable)
Memory at fd5fe000 (32-bit, non-prefetchable)
I/O ports at 4000
Capabilities: [40] Power Management version 3
Capabilities: [48] Express Endpoint, MSI 00
Capabilities: [84] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [9c] MSI-X: Enable+ Count=65 Masked-
# ifconfig
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vmxnet3s1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.50 netmask ffffff00 broadcast 192.168.1.255
ether 0:c:29:34:3c:79
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
vmxnet3s1: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 1500 index 2
inet6 ::/0
ether 0:c:29:34:3c:79Here's the FreeBSD 12.2-RELEASE
iperf3 scores:
Code:
# iperf3 -c 192.168.1.71
Connecting to host 192.168.1.71, port 5201
[ 4] local 192.168.1.50 port 48305 connected to 192.168.1.71 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 345 MBytes 2.89 Gbits/sec
[ 4] 1.00-2.00 sec 392 MBytes 3.29 Gbits/sec
[ 4] 2.00-3.00 sec 239 MBytes 2.01 Gbits/sec
[ 4] 3.00-4.00 sec 392 MBytes 3.29 Gbits/sec
[ 4] 4.00-5.00 sec 424 MBytes 3.55 Gbits/sec
[ 4] 5.00-6.00 sec 427 MBytes 3.58 Gbits/sec
[ 4] 6.00-7.00 sec 412 MBytes 3.46 Gbits/sec
[ 4] 7.00-8.00 sec 364 MBytes 3.05 Gbits/sec
[ 4] 8.00-9.00 sec 234 MBytes 1.96 Gbits/sec
[ 4] 9.00-10.00 sec 300 MBytes 2.51 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.45 GBytes 2.96 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.44 GBytes 2.96 Gbits/sec receiverHere's the Ubuntu 20.04 VM's
iperf3 scores:
Code:
# iperf3 -c 192.168.1.28
Connecting to host 192.168.1.28, port 5201
[ 4] local 192.168.1.50 port 50214 connected to 192.168.1.28 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.02 GBytes 8.72 Gbits/sec
[ 4] 1.00-2.00 sec 1.09 GBytes 9.33 Gbits/sec
[ 4] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 4] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec
[ 4] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec
[ 4] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 7.00-8.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 4] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.9 GBytes 9.33 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.9 GBytes 9.33 Gbits/sec receiverOPNsense, which is based on FreeBSD, also has poor 10GbE performance. But tests show versions of OPNSense based on FreeBSD 11 had better 10GbE throughput than FreeBSD 12.
Is this throughput discrepancy a regression in FreeBSD? Can it be fixed by end user, or is it something that must be accomplished upstream?
Sources:
Poor Throughput (Even On Same Network Segment)
Poor Throughput (Even On Same Network Segment)
forum.opnsense.org
Poor Throughput (Even On Same Network Segment)
Poor Throughput (Even On Same Network Segment)
forum.opnsense.org
Update: Some other interesting observations ---
FreeBSD
iperf3 "self test":
Code:
# iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 58398 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.63 GBytes 31.2 Gbits/sec 0 861 KBytes
[ 5] 1.00-2.00 sec 4.09 GBytes 35.2 Gbits/sec 0 1.62 MBytes
[ 5] 2.00-3.00 sec 4.01 GBytes 34.5 Gbits/sec 0 1.77 MBytes
[ 5] 3.00-4.00 sec 3.97 GBytes 34.1 Gbits/sec 0 1.77 MBytes
[ 5] 4.00-5.00 sec 4.19 GBytes 36.0 Gbits/sec 0 1.77 MBytes
[ 5] 5.00-6.00 sec 3.92 GBytes 33.7 Gbits/sec 0 1.77 MBytes
[ 5] 6.00-7.00 sec 3.96 GBytes 34.0 Gbits/sec 0 1.77 MBytes
[ 5] 7.00-8.00 sec 4.20 GBytes 36.1 Gbits/sec 0 1.77 MBytes
[ 5] 8.00-9.00 sec 4.07 GBytes 34.9 Gbits/sec 0 1.77 MBytes
[ 5] 9.00-10.00 sec 4.03 GBytes 34.6 Gbits/sec 0 1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 40.1 GBytes 34.4 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 40.1 GBytes 34.4 Gbits/sec receiverUbuntu 20.04
iperf3 "self test":
Code:
# iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 45130 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.40 GBytes 29.2 Gbits/sec 0 1.25 MBytes
[ 5] 1.00-2.00 sec 3.39 GBytes 29.1 Gbits/sec 0 1.25 MBytes
[ 5] 2.00-3.00 sec 3.46 GBytes 29.7 Gbits/sec 0 1.25 MBytes
[ 5] 3.00-4.00 sec 3.48 GBytes 29.9 Gbits/sec 0 1.31 MBytes
[ 5] 4.00-5.00 sec 3.50 GBytes 30.1 Gbits/sec 0 1.31 MBytes
[ 5] 5.00-6.00 sec 3.40 GBytes 29.2 Gbits/sec 0 1.31 MBytes
[ 5] 6.00-7.00 sec 3.39 GBytes 29.1 Gbits/sec 0 1.75 MBytes
[ 5] 7.00-8.00 sec 3.46 GBytes 29.7 Gbits/sec 0 1.75 MBytes
[ 5] 8.00-9.00 sec 3.45 GBytes 29.7 Gbits/sec 0 1.75 MBytes
[ 5] 9.00-10.00 sec 3.44 GBytes 29.6 Gbits/sec 0 2.81 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 34.4 GBytes 29.5 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 34.4 GBytes 29.5 Gbits/sec receiverFreeBSD to Ubuntu 20.04
iperf3 test (same host):
Code:
# iperf3 -c 192.168.1.28
Connecting to host 192.168.1.28, port 5201
[ 5] local 192.168.1.71 port 42002 connected to 192.168.1.28 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.49 GBytes 21.4 Gbits/sec 0 1.03 MBytes
[ 5] 1.00-2.00 sec 2.84 GBytes 24.4 Gbits/sec 0 2.98 MBytes
[ 5] 2.00-3.00 sec 2.86 GBytes 24.6 Gbits/sec 0 2.99 MBytes
[ 5] 3.00-4.00 sec 2.84 GBytes 24.4 Gbits/sec 0 2.99 MBytes
[ 5] 4.00-5.00 sec 2.92 GBytes 25.1 Gbits/sec 0 2.99 MBytes
[ 5] 5.00-6.00 sec 2.85 GBytes 24.5 Gbits/sec 0 2.99 MBytes
[ 5] 6.00-7.00 sec 2.86 GBytes 24.5 Gbits/sec 0 2.99 MBytes
[ 5] 7.00-8.00 sec 2.77 GBytes 23.8 Gbits/sec 0 2.99 MBytes
[ 5] 8.00-9.00 sec 2.98 GBytes 25.6 Gbits/sec 0 3.00 MBytes
[ 5] 9.00-10.00 sec 2.81 GBytes 24.1 Gbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 28.2 GBytes 24.2 Gbits/sec 0 sender
[ 5] 0.00-10.19 sec 28.2 GBytes 23.8 Gbits/sec receiverUbuntu 20.04 to FreeBSD
iperf3 test (same host):
Code:
# iperf3 -c 192.168.1.71
Connecting to host 192.168.1.71, port 5201
[ 5] local 192.168.1.28 port 58162 connected to 192.168.1.71 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 370 MBytes 3.11 Gbits/sec 9 625 KBytes
[ 5] 1.00-2.00 sec 405 MBytes 3.40 Gbits/sec 5 655 KBytes
[ 5] 2.00-3.00 sec 396 MBytes 3.32 Gbits/sec 19 693 KBytes
[ 5] 3.00-4.00 sec 381 MBytes 3.20 Gbits/sec 12 706 KBytes
[ 5] 4.00-5.00 sec 391 MBytes 3.28 Gbits/sec 21 519 KBytes
[ 5] 5.00-6.00 sec 404 MBytes 3.39 Gbits/sec 14 591 KBytes
[ 5] 6.00-7.00 sec 392 MBytes 3.29 Gbits/sec 5 634 KBytes
[ 5] 7.00-8.00 sec 374 MBytes 3.14 Gbits/sec 10 672 KBytes
[ 5] 8.00-9.00 sec 364 MBytes 3.05 Gbits/sec 9 662 KBytes
[ 5] 9.00-10.00 sec 384 MBytes 3.22 Gbits/sec 11 700 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.77 GBytes 3.24 Gbits/sec 115 sender
[ 5] 0.00-10.00 sec 3.77 GBytes 3.24 Gbits/sec receiverWow...
Last edited: