Hello,
I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3.
The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)
They are connected to a Dell N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc.
When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in https://calomel.org/freebsd_network_tuning.html with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput).
The same exact servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run.
I have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce.
I have even tried changing to the RACK TCP stack, only to get slightly better results, however with RACK the amount of retries is nearly 0.
Also, the throughput seems to be very sporadic, here are two tests one after another.
Thank You.
I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3.
The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)
They are connected to a Dell N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc.
When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in https://calomel.org/freebsd_network_tuning.html with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput).
The same exact servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run.
I have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce.
Code:
# ifconfig mce0
mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ffed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT>
ether b8:ce:f6:81:df:6a
inet 192.168.10.31 netmask 0xffffff00 broadcast 192.168.10.255
media: Ethernet 25GBase-CR <full-duplex,rxpause,txpause>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Code:
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[ 5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 930 MBytes 7.80 Gbits/sec 62 789 KBytes
[ 5] 1.00-2.00 sec 942 MBytes 7.90 Gbits/sec 164 824 KBytes
[ 5] 2.00-3.00 sec 1.00 GBytes 8.61 Gbits/sec 402 879 KBytes
[ 5] 3.00-4.00 sec 761 MBytes 6.39 Gbits/sec 61 588 KBytes
[ 5] 4.00-5.00 sec 724 MBytes 6.07 Gbits/sec 220 497 KBytes
[ 5] 5.00-6.00 sec 723 MBytes 6.07 Gbits/sec 54 364 KBytes
[ 5] 6.00-7.00 sec 716 MBytes 6.01 Gbits/sec 187 682 KBytes
[ 5] 7.00-8.00 sec 728 MBytes 6.11 Gbits/sec 86 568 KBytes
[ 5] 8.00-9.00 sec 761 MBytes 6.39 Gbits/sec 37 418 KBytes
[ 5] 9.00-10.00 sec 733 MBytes 6.15 Gbits/sec 8 617 KBytes
[ 5] 10.00-11.00 sec 734 MBytes 6.16 Gbits/sec 238 474 KBytes
[ 5] 11.00-12.00 sec 736 MBytes 6.17 Gbits/sec 164 757 KBytes
[ 5] 12.00-13.00 sec 610 MBytes 5.12 Gbits/sec 118 579 KBytes
[ 5] 13.00-14.00 sec 1.02 GBytes 8.75 Gbits/sec 447 449 KBytes
[ 5] 14.00-15.00 sec 728 MBytes 6.11 Gbits/sec 132 719 KBytes
[ 5] 15.00-16.00 sec 724 MBytes 6.07 Gbits/sec 185 649 KBytes
[ 5] 16.00-17.00 sec 597 MBytes 5.01 Gbits/sec 142 570 KBytes
[ 5] 17.00-18.00 sec 733 MBytes 6.15 Gbits/sec 102 484 KBytes
[ 5] 18.00-19.00 sec 726 MBytes 6.09 Gbits/sec 15 569 KBytes
[ 5] 19.00-20.00 sec 733 MBytes 6.15 Gbits/sec 181 527 KBytes
[ 5] 20.00-21.00 sec 729 MBytes 6.12 Gbits/sec 118 430 KBytes
[ 5] 21.00-22.00 sec 733 MBytes 6.15 Gbits/sec 116 641 KBytes
[ 5] 22.00-23.00 sec 728 MBytes 6.10 Gbits/sec 182 598 KBytes
[ 5] 23.00-24.00 sec 743 MBytes 6.24 Gbits/sec 209 614 KBytes
[ 5] 24.00-25.00 sec 746 MBytes 6.26 Gbits/sec 72 758 KBytes
[ 5] 25.00-26.00 sec 742 MBytes 6.23 Gbits/sec 199 675 KBytes
[ 5] 26.00-27.00 sec 799 MBytes 6.70 Gbits/sec 183 542 KBytes
[ 5] 27.00-28.00 sec 908 MBytes 7.61 Gbits/sec 7 1.19 MBytes
[ 5] 28.00-29.00 sec 1.37 GBytes 11.7 Gbits/sec 606 1013 KBytes
[ 5] 29.00-30.00 sec 1.31 GBytes 11.3 Gbits/sec 74 1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 23.7 GBytes 6.79 Gbits/sec 4771 sender
[ 5] 0.00-30.00 sec 23.7 GBytes 6.79 Gbits/sec receiver
I have even tried changing to the RACK TCP stack, only to get slightly better results, however with RACK the amount of retries is nearly 0.
Code:
[root@db-02 ~]# sysctl net.inet.tcp.functions_default=rack
net.inet.tcp.functions_default: rack -> rack
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[ 5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 761 MBytes 6.38 Gbits/sec 0 737 KBytes
[ 5] 1.00-2.00 sec 859 MBytes 7.21 Gbits/sec 0 761 KBytes
[ 5] 2.00-3.00 sec 880 MBytes 7.38 Gbits/sec 0 785 KBytes
[ 5] 3.00-4.00 sec 734 MBytes 6.16 Gbits/sec 0 804 KBytes
[ 5] 4.00-5.00 sec 777 MBytes 6.52 Gbits/sec 0 824 KBytes
[ 5] 5.00-6.00 sec 719 MBytes 6.03 Gbits/sec 0 841 KBytes
[ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 862 KBytes
[ 5] 7.00-8.00 sec 880 MBytes 7.38 Gbits/sec 0 882 KBytes
[ 5] 8.00-9.00 sec 906 MBytes 7.60 Gbits/sec 0 904 KBytes
[ 5] 9.00-10.00 sec 749 MBytes 6.29 Gbits/sec 0 921 KBytes
[ 5] 10.00-11.00 sec 798 MBytes 6.69 Gbits/sec 0 938 KBytes
[ 5] 11.00-12.00 sec 746 MBytes 6.26 Gbits/sec 209 772 KBytes
[ 5] 12.00-13.00 sec 768 MBytes 6.44 Gbits/sec 35 644 KBytes
[ 5] 13.00-14.00 sec 948 MBytes 7.95 Gbits/sec 0 673 KBytes
[ 5] 14.00-15.00 sec 1.23 GBytes 10.5 Gbits/sec 0 711 KBytes
[ 5] 15.00-16.00 sec 1.32 GBytes 11.4 Gbits/sec 0 748 KBytes
[ 5] 16.00-17.00 sec 1.31 GBytes 11.2 Gbits/sec 0 785 KBytes
[ 5] 17.00-18.00 sec 1.29 GBytes 11.1 Gbits/sec 0 819 KBytes
[ 5] 18.00-19.00 sec 1.30 GBytes 11.2 Gbits/sec 0 852 KBytes
[ 5] 19.00-20.00 sec 1.34 GBytes 11.5 Gbits/sec 0 883 KBytes
[ 5] 20.00-21.00 sec 1.29 GBytes 11.1 Gbits/sec 0 914 KBytes
[ 5] 21.00-22.00 sec 1.36 GBytes 11.7 Gbits/sec 0 944 KBytes
[ 5] 22.00-23.00 sec 1.33 GBytes 11.4 Gbits/sec 0 974 KBytes
[ 5] 23.00-24.00 sec 1.31 GBytes 11.2 Gbits/sec 0 1003 KBytes
[ 5] 24.00-25.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.00 MBytes
[ 5] 25.00-26.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.03 MBytes
[ 5] 26.00-27.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.06 MBytes
[ 5] 27.00-28.00 sec 957 MBytes 8.03 Gbits/sec 0 1.07 MBytes
[ 5] 28.00-29.00 sec 837 MBytes 7.02 Gbits/sec 0 1.09 MBytes
[ 5] 29.00-30.00 sec 729 MBytes 6.11 Gbits/sec 0 1.10 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 30.6 GBytes 8.77 Gbits/sec 244 sender
[ 5] 0.00-30.00 sec 30.6 GBytes 8.77 Gbits/sec receiver
Also, the throughput seems to be very sporadic, here are two tests one after another.
Code:
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[ 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.47 GBytes 12.6 Gbits/sec 174 983 KBytes
[ 5] 1.00-2.00 sec 1.53 GBytes 13.1 Gbits/sec 0 1.49 MBytes
[ 5] 2.00-3.00 sec 1.23 GBytes 10.6 Gbits/sec 0 1.60 MBytes
[ 5] 3.00-4.00 sec 1.51 GBytes 13.0 Gbits/sec 0 1.60 MBytes
[ 5] 4.00-5.00 sec 1.99 GBytes 17.1 Gbits/sec 287 928 KBytes
[ 5] 5.00-6.00 sec 1.48 GBytes 12.7 Gbits/sec 630 505 KBytes
[ 5] 6.00-7.00 sec 1.00 GBytes 8.60 Gbits/sec 471 1.03 MBytes
[ 5] 7.00-8.00 sec 1.00 GBytes 8.61 Gbits/sec 434 745 KBytes
[ 5] 8.00-9.00 sec 1.03 GBytes 8.86 Gbits/sec 439 402 KBytes
[ 5] 9.00-10.00 sec 910 MBytes 7.63 Gbits/sec 157 657 KBytes
[ 5] 10.00-11.00 sec 1.25 GBytes 10.7 Gbits/sec 78 487 KBytes
[ 5] 11.00-12.00 sec 1.41 GBytes 12.1 Gbits/sec 567 1.21 MBytes
[ 5] 12.00-13.00 sec 1.42 GBytes 12.2 Gbits/sec 45 1.41 MBytes
[ 5] 13.00-14.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.60 MBytes
[ 5] 14.00-15.00 sec 1.26 GBytes 10.8 Gbits/sec 44 444 KBytes
[ 5] 15.00-16.00 sec 682 MBytes 5.72 Gbits/sec 195 390 KBytes
[ 5] 16.00-17.00 sec 897 MBytes 7.52 Gbits/sec 298 467 KBytes
[ 5] 17.00-18.00 sec 884 MBytes 7.42 Gbits/sec 171 658 KBytes
[ 5] 18.00-19.00 sec 1.24 GBytes 10.6 Gbits/sec 195 1.24 MBytes
[ 5] 19.00-20.00 sec 1.32 GBytes 11.4 Gbits/sec 189 1.21 MBytes
[ 5] 20.00-21.00 sec 826 MBytes 6.93 Gbits/sec 398 462 KBytes
[ 5] 21.00-22.00 sec 600 MBytes 5.03 Gbits/sec 20 268 KBytes
[ 5] 22.00-23.00 sec 621 MBytes 5.21 Gbits/sec 22 673 KBytes
[ 5] 23.00-24.00 sec 656 MBytes 5.51 Gbits/sec 43 540 KBytes
[ 5] 24.00-25.00 sec 669 MBytes 5.61 Gbits/sec 70 488 KBytes
[ 5] 25.00-26.00 sec 664 MBytes 5.57 Gbits/sec 34 251 KBytes
[ 5] 26.00-27.00 sec 669 MBytes 5.61 Gbits/sec 78 515 KBytes
[ 5] 27.00-28.00 sec 666 MBytes 5.59 Gbits/sec 33 421 KBytes
[ 5] 28.00-29.00 sec 661 MBytes 5.54 Gbits/sec 167 309 KBytes
[ 5] 29.00-30.00 sec 666 MBytes 5.59 Gbits/sec 27 611 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 31.3 GBytes 8.98 Gbits/sec 5266 sender
[ 5] 0.00-30.00 sec 31.3 GBytes 8.98 Gbits/sec receiver
iperf Done.
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[ 5] local 192.168.10.31 port 51193 connected to 192.168.10.30 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 663 MBytes 5.56 Gbits/sec 297 411 KBytes
[ 5] 1.00-2.00 sec 725 MBytes 6.08 Gbits/sec 394 795 KBytes
[ 5] 2.00-3.00 sec 722 MBytes 6.06 Gbits/sec 514 403 KBytes
[ 5] 3.00-4.00 sec 696 MBytes 5.84 Gbits/sec 49 716 KBytes
[ 5] 4.00-5.00 sec 699 MBytes 5.86 Gbits/sec 54 405 KBytes
[ 5] 5.00-6.00 sec 796 MBytes 6.68 Gbits/sec 171 684 KBytes
[ 5] 6.00-7.00 sec 1.06 GBytes 9.08 Gbits/sec 28 596 KBytes
[ 5] 7.00-8.00 sec 341 MBytes 2.86 Gbits/sec 129 748 KBytes
[ 5] 8.00-9.00 sec 726 MBytes 6.09 Gbits/sec 203 657 KBytes
[ 5] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec 330 269 KBytes
[ 5] 10.00-11.00 sec 683 MBytes 5.73 Gbits/sec 265 521 KBytes
[ 5] 11.00-12.00 sec 682 MBytes 5.72 Gbits/sec 41 487 KBytes
[ 5] 12.00-13.00 sec 683 MBytes 5.73 Gbits/sec 96 697 KBytes
[ 5] 13.00-14.00 sec 657 MBytes 5.51 Gbits/sec 134 685 KBytes
[ 5] 14.00-15.00 sec 658 MBytes 5.52 Gbits/sec 64 434 KBytes
[ 5] 15.00-16.00 sec 656 MBytes 5.51 Gbits/sec 164 616 KBytes
[ 5] 16.00-17.00 sec 662 MBytes 5.55 Gbits/sec 98 465 KBytes
[ 5] 17.00-18.00 sec 661 MBytes 5.54 Gbits/sec 83 512 KBytes
[ 5] 18.00-19.00 sec 653 MBytes 5.47 Gbits/sec 120 244 KBytes
[ 5] 19.00-20.00 sec 657 MBytes 5.51 Gbits/sec 157 660 KBytes
[ 5] 20.00-21.00 sec 649 MBytes 5.45 Gbits/sec 99 643 KBytes
[ 5] 21.00-22.00 sec 752 MBytes 6.31 Gbits/sec 57 371 KBytes
[ 5] 22.00-23.00 sec 688 MBytes 5.77 Gbits/sec 160 666 KBytes
[ 5] 23.00-24.00 sec 697 MBytes 5.85 Gbits/sec 195 280 KBytes
[ 5] 24.00-25.00 sec 697 MBytes 5.85 Gbits/sec 90 724 KBytes
[ 5] 25.00-26.00 sec 696 MBytes 5.84 Gbits/sec 195 514 KBytes
[ 5] 26.00-27.00 sec 692 MBytes 5.80 Gbits/sec 56 498 KBytes
[ 5] 27.00-28.00 sec 693 MBytes 5.81 Gbits/sec 143 537 KBytes
[ 5] 28.00-29.00 sec 704 MBytes 5.90 Gbits/sec 107 643 KBytes
[ 5] 29.00-30.00 sec 704 MBytes 5.91 Gbits/sec 135 675 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 20.3 GBytes 5.80 Gbits/sec 4628 sender
[ 5] 0.00-30.00 sec 20.3 GBytes 5.80 Gbits/sec receiver
Thank You.