Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[mike.jakubik]

Hello,

I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, however, I am unable to get much more than 6Gb/s when testing with iperf3.

The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)

They are connected to a Dell N3224PX-ON switch. Both servers are idle and not in use, with a fresh install of stable/13-ebea872f8, nothing running on them except ssh, sendmail, etc.

When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I have tried just about every knob listed in https://calomel.org/freebsd_network_tuning.html with little impact on the performance. The network cards have HW LRO enabled as per the driver documentation (though this only seems to lower IRQ usage with no impact on actual throughput).

The same exact servers tested on Linux (fedora 34) produced nearly 3x the performance (see attached screenshots), i was able to get a steady 14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to avg at around 6Gbs but it is very sporadic during the iperf run.

I have run out of ideas, any suggestions are welcome. Considering Netflix uses very similar HW and they push 400 Gb/s tells me there is something really wrong here or Netflix isnt sharing all their secret sauce.

# ifconfig mce0
mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ffed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT>
        ether b8:ce:f6:81:df:6a
        inet 192.168.10.31 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet 25GBase-CR <full-duplex,rxpause,txpause>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   930 MBytes  7.80 Gbits/sec   62    789 KBytes
[  5]   1.00-2.00   sec   942 MBytes  7.90 Gbits/sec  164    824 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.61 Gbits/sec  402    879 KBytes
[  5]   3.00-4.00   sec   761 MBytes  6.39 Gbits/sec   61    588 KBytes
[  5]   4.00-5.00   sec   724 MBytes  6.07 Gbits/sec  220    497 KBytes
[  5]   5.00-6.00   sec   723 MBytes  6.07 Gbits/sec   54    364 KBytes
[  5]   6.00-7.00   sec   716 MBytes  6.01 Gbits/sec  187    682 KBytes
[  5]   7.00-8.00   sec   728 MBytes  6.11 Gbits/sec   86    568 KBytes
[  5]   8.00-9.00   sec   761 MBytes  6.39 Gbits/sec   37    418 KBytes
[  5]   9.00-10.00  sec   733 MBytes  6.15 Gbits/sec    8    617 KBytes
[  5]  10.00-11.00  sec   734 MBytes  6.16 Gbits/sec  238    474 KBytes
[  5]  11.00-12.00  sec   736 MBytes  6.17 Gbits/sec  164    757 KBytes
[  5]  12.00-13.00  sec   610 MBytes  5.12 Gbits/sec  118    579 KBytes
[  5]  13.00-14.00  sec  1.02 GBytes  8.75 Gbits/sec  447    449 KBytes
[  5]  14.00-15.00  sec   728 MBytes  6.11 Gbits/sec  132    719 KBytes
[  5]  15.00-16.00  sec   724 MBytes  6.07 Gbits/sec  185    649 KBytes
[  5]  16.00-17.00  sec   597 MBytes  5.01 Gbits/sec  142    570 KBytes
[  5]  17.00-18.00  sec   733 MBytes  6.15 Gbits/sec  102    484 KBytes
[  5]  18.00-19.00  sec   726 MBytes  6.09 Gbits/sec   15    569 KBytes
[  5]  19.00-20.00  sec   733 MBytes  6.15 Gbits/sec  181    527 KBytes
[  5]  20.00-21.00  sec   729 MBytes  6.12 Gbits/sec  118    430 KBytes
[  5]  21.00-22.00  sec   733 MBytes  6.15 Gbits/sec  116    641 KBytes
[  5]  22.00-23.00  sec   728 MBytes  6.10 Gbits/sec  182    598 KBytes
[  5]  23.00-24.00  sec   743 MBytes  6.24 Gbits/sec  209    614 KBytes
[  5]  24.00-25.00  sec   746 MBytes  6.26 Gbits/sec   72    758 KBytes
[  5]  25.00-26.00  sec   742 MBytes  6.23 Gbits/sec  199    675 KBytes
[  5]  26.00-27.00  sec   799 MBytes  6.70 Gbits/sec  183    542 KBytes
[  5]  27.00-28.00  sec   908 MBytes  7.61 Gbits/sec    7   1.19 MBytes
[  5]  28.00-29.00  sec  1.37 GBytes  11.7 Gbits/sec  606   1013 KBytes
[  5]  29.00-30.00  sec  1.31 GBytes  11.3 Gbits/sec   74   1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  23.7 GBytes  6.79 Gbits/sec  4771             sender
[  5]   0.00-30.00  sec  23.7 GBytes  6.79 Gbits/sec                  receiver

I have even tried changing to the RACK TCP stack, only to get slightly better results, however with RACK the amount of retries is nearly 0.

[root@db-02 ~]# sysctl net.inet.tcp.functions_default=rack
net.inet.tcp.functions_default: rack -> rack
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   761 MBytes  6.38 Gbits/sec    0    737 KBytes
[  5]   1.00-2.00   sec   859 MBytes  7.21 Gbits/sec    0    761 KBytes
[  5]   2.00-3.00   sec   880 MBytes  7.38 Gbits/sec    0    785 KBytes
[  5]   3.00-4.00   sec   734 MBytes  6.16 Gbits/sec    0    804 KBytes
[  5]   4.00-5.00   sec   777 MBytes  6.52 Gbits/sec    0    824 KBytes
[  5]   5.00-6.00   sec   719 MBytes  6.03 Gbits/sec    0    841 KBytes
[  5]   6.00-7.00   sec   865 MBytes  7.26 Gbits/sec    0    862 KBytes
[  5]   7.00-8.00   sec   880 MBytes  7.38 Gbits/sec    0    882 KBytes
[  5]   8.00-9.00   sec   906 MBytes  7.60 Gbits/sec    0    904 KBytes
[  5]   9.00-10.00  sec   749 MBytes  6.29 Gbits/sec    0    921 KBytes
[  5]  10.00-11.00  sec   798 MBytes  6.69 Gbits/sec    0    938 KBytes
[  5]  11.00-12.00  sec   746 MBytes  6.26 Gbits/sec  209    772 KBytes
[  5]  12.00-13.00  sec   768 MBytes  6.44 Gbits/sec   35    644 KBytes
[  5]  13.00-14.00  sec   948 MBytes  7.95 Gbits/sec    0    673 KBytes
[  5]  14.00-15.00  sec  1.23 GBytes  10.5 Gbits/sec    0    711 KBytes
[  5]  15.00-16.00  sec  1.32 GBytes  11.4 Gbits/sec    0    748 KBytes
[  5]  16.00-17.00  sec  1.31 GBytes  11.2 Gbits/sec    0    785 KBytes
[  5]  17.00-18.00  sec  1.29 GBytes  11.1 Gbits/sec    0    819 KBytes
[  5]  18.00-19.00  sec  1.30 GBytes  11.2 Gbits/sec    0    852 KBytes
[  5]  19.00-20.00  sec  1.34 GBytes  11.5 Gbits/sec    0    883 KBytes
[  5]  20.00-21.00  sec  1.29 GBytes  11.1 Gbits/sec    0    914 KBytes
[  5]  21.00-22.00  sec  1.36 GBytes  11.7 Gbits/sec    0    944 KBytes
[  5]  22.00-23.00  sec  1.33 GBytes  11.4 Gbits/sec    0    974 KBytes
[  5]  23.00-24.00  sec  1.31 GBytes  11.2 Gbits/sec    0   1003 KBytes
[  5]  24.00-25.00  sec  1.30 GBytes  11.2 Gbits/sec    0   1.00 MBytes
[  5]  25.00-26.00  sec  1.34 GBytes  11.5 Gbits/sec    0   1.03 MBytes
[  5]  26.00-27.00  sec  1.32 GBytes  11.3 Gbits/sec    0   1.06 MBytes
[  5]  27.00-28.00  sec   957 MBytes  8.03 Gbits/sec    0   1.07 MBytes
[  5]  28.00-29.00  sec   837 MBytes  7.02 Gbits/sec    0   1.09 MBytes
[  5]  29.00-30.00  sec   729 MBytes  6.11 Gbits/sec    0   1.10 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  30.6 GBytes  8.77 Gbits/sec  244             sender
[  5]   0.00-30.00  sec  30.6 GBytes  8.77 Gbits/sec                  receiver

Also, the throughput seems to be very sporadic, here are two tests one after another.

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.47 GBytes  12.6 Gbits/sec  174    983 KBytes
[  5]   1.00-2.00   sec  1.53 GBytes  13.1 Gbits/sec    0   1.49 MBytes
[  5]   2.00-3.00   sec  1.23 GBytes  10.6 Gbits/sec    0   1.60 MBytes
[  5]   3.00-4.00   sec  1.51 GBytes  13.0 Gbits/sec    0   1.60 MBytes
[  5]   4.00-5.00   sec  1.99 GBytes  17.1 Gbits/sec  287    928 KBytes
[  5]   5.00-6.00   sec  1.48 GBytes  12.7 Gbits/sec  630    505 KBytes
[  5]   6.00-7.00   sec  1.00 GBytes  8.60 Gbits/sec  471   1.03 MBytes
[  5]   7.00-8.00   sec  1.00 GBytes  8.61 Gbits/sec  434    745 KBytes
[  5]   8.00-9.00   sec  1.03 GBytes  8.86 Gbits/sec  439    402 KBytes
[  5]   9.00-10.00  sec   910 MBytes  7.63 Gbits/sec  157    657 KBytes
[  5]  10.00-11.00  sec  1.25 GBytes  10.7 Gbits/sec   78    487 KBytes
[  5]  11.00-12.00  sec  1.41 GBytes  12.1 Gbits/sec  567   1.21 MBytes
[  5]  12.00-13.00  sec  1.42 GBytes  12.2 Gbits/sec   45   1.41 MBytes
[  5]  13.00-14.00  sec  1.37 GBytes  11.7 Gbits/sec    0   1.60 MBytes
[  5]  14.00-15.00  sec  1.26 GBytes  10.8 Gbits/sec   44    444 KBytes
[  5]  15.00-16.00  sec   682 MBytes  5.72 Gbits/sec  195    390 KBytes
[  5]  16.00-17.00  sec   897 MBytes  7.52 Gbits/sec  298    467 KBytes
[  5]  17.00-18.00  sec   884 MBytes  7.42 Gbits/sec  171    658 KBytes
[  5]  18.00-19.00  sec  1.24 GBytes  10.6 Gbits/sec  195   1.24 MBytes
[  5]  19.00-20.00  sec  1.32 GBytes  11.4 Gbits/sec  189   1.21 MBytes
[  5]  20.00-21.00  sec   826 MBytes  6.93 Gbits/sec  398    462 KBytes
[  5]  21.00-22.00  sec   600 MBytes  5.03 Gbits/sec   20    268 KBytes
[  5]  22.00-23.00  sec   621 MBytes  5.21 Gbits/sec   22    673 KBytes
[  5]  23.00-24.00  sec   656 MBytes  5.51 Gbits/sec   43    540 KBytes
[  5]  24.00-25.00  sec   669 MBytes  5.61 Gbits/sec   70    488 KBytes
[  5]  25.00-26.00  sec   664 MBytes  5.57 Gbits/sec   34    251 KBytes
[  5]  26.00-27.00  sec   669 MBytes  5.61 Gbits/sec   78    515 KBytes
[  5]  27.00-28.00  sec   666 MBytes  5.59 Gbits/sec   33    421 KBytes
[  5]  28.00-29.00  sec   661 MBytes  5.54 Gbits/sec  167    309 KBytes
[  5]  29.00-30.00  sec   666 MBytes  5.59 Gbits/sec   27    611 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  31.3 GBytes  8.98 Gbits/sec  5266             sender
[  5]   0.00-30.00  sec  31.3 GBytes  8.98 Gbits/sec                  receiver

iperf Done.
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168.10.31 port 51193 connected to 192.168.10.30 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   663 MBytes  5.56 Gbits/sec  297    411 KBytes
[  5]   1.00-2.00   sec   725 MBytes  6.08 Gbits/sec  394    795 KBytes
[  5]   2.00-3.00   sec   722 MBytes  6.06 Gbits/sec  514    403 KBytes
[  5]   3.00-4.00   sec   696 MBytes  5.84 Gbits/sec   49    716 KBytes
[  5]   4.00-5.00   sec   699 MBytes  5.86 Gbits/sec   54    405 KBytes
[  5]   5.00-6.00   sec   796 MBytes  6.68 Gbits/sec  171    684 KBytes
[  5]   6.00-7.00   sec  1.06 GBytes  9.08 Gbits/sec   28    596 KBytes
[  5]   7.00-8.00   sec   341 MBytes  2.86 Gbits/sec  129    748 KBytes
[  5]   8.00-9.00   sec   726 MBytes  6.09 Gbits/sec  203    657 KBytes
[  5]   9.00-10.00  sec   682 MBytes  5.72 Gbits/sec  330    269 KBytes
[  5]  10.00-11.00  sec   683 MBytes  5.73 Gbits/sec  265    521 KBytes
[  5]  11.00-12.00  sec   682 MBytes  5.72 Gbits/sec   41    487 KBytes
[  5]  12.00-13.00  sec   683 MBytes  5.73 Gbits/sec   96    697 KBytes
[  5]  13.00-14.00  sec   657 MBytes  5.51 Gbits/sec  134    685 KBytes
[  5]  14.00-15.00  sec   658 MBytes  5.52 Gbits/sec   64    434 KBytes
[  5]  15.00-16.00  sec   656 MBytes  5.51 Gbits/sec  164    616 KBytes
[  5]  16.00-17.00  sec   662 MBytes  5.55 Gbits/sec   98    465 KBytes
[  5]  17.00-18.00  sec   661 MBytes  5.54 Gbits/sec   83    512 KBytes
[  5]  18.00-19.00  sec   653 MBytes  5.47 Gbits/sec  120    244 KBytes
[  5]  19.00-20.00  sec   657 MBytes  5.51 Gbits/sec  157    660 KBytes
[  5]  20.00-21.00  sec   649 MBytes  5.45 Gbits/sec   99    643 KBytes
[  5]  21.00-22.00  sec   752 MBytes  6.31 Gbits/sec   57    371 KBytes
[  5]  22.00-23.00  sec   688 MBytes  5.77 Gbits/sec  160    666 KBytes
[  5]  23.00-24.00  sec   697 MBytes  5.85 Gbits/sec  195    280 KBytes
[  5]  24.00-25.00  sec   697 MBytes  5.85 Gbits/sec   90    724 KBytes
[  5]  25.00-26.00  sec   696 MBytes  5.84 Gbits/sec  195    514 KBytes
[  5]  26.00-27.00  sec   692 MBytes  5.80 Gbits/sec   56    498 KBytes
[  5]  27.00-28.00  sec   693 MBytes  5.81 Gbits/sec  143    537 KBytes
[  5]  28.00-29.00  sec   704 MBytes  5.90 Gbits/sec  107    643 KBytes
[  5]  29.00-30.00  sec   704 MBytes  5.91 Gbits/sec  135    675 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  20.3 GBytes  5.80 Gbits/sec  4628             sender
[  5]   0.00-30.00  sec  20.3 GBytes  5.80 Gbits/sec                  receiver

Thank You.

 

[mike.jakubik]

Here is also a graph screenshot of a 300 second run, notice the fluctuation between 5 - 10 Gb/s. Also, the CPU stays at about 1% intr and 1.5 avg system load.

 

[jbo@]

Hello & Welcome to this FreeBSD community!

Unfortunately, I don't have a direct answer for you. However, I can share what I'd be doing in your position (which may or may not be helpful).

Have you tried increasing the transfer size -n option? 1 GB is not that much data over a 25Gbps link:

-n, --bytes n[KMGT]
number of bytes to transmit (instead of -t)

Have you tried to increase the size of the read/write buffer -l option?

-l, --length n[KMGT]
length of buffer to read or write. For TCP tests, the default
value is 128KB. In the case of UDP, iperf3 tries to dynamically
determine a reasonable sending size based on the path MTU; if
that cannot be determined it uses 1460 bytes as a sending size.
For SCTP tests, the default size is 64KB.

These are some of the things I'd be testing to get some more data on the behavior of the system.
I had a similar situation in the past (although with a 10 Gbps system). Just piping a huge file through the connection instead of using iperf3(1) yielded different results.

Other than that, is it possible that you're running a kernel with various debug knobs enable? This can drastically lower performance. I'd have to check the default settings but these come to mind:

nooptions       INVARIANTS
nooptions       INVARIANT_SUPPORT
nooptions       WITNESS
nooptions       WITNESS_SKIPSPIN
nooptions       DEBUG_VFS_LOCKS
nooptions       BUF_TRACKING
nooptions       FULL_BUF_TRACKING
nooptions       DEADLKRES
nooptions       COVERAGE
nooptions       KCOV
nooptions       MALLOC_DEBUG_MAXZONES
nooptions       QUEUE_MACRO_DEBUG_TRASH

nooptions       IEEE80211_DEBUG

If you could also share the iperf3(1) runs on Linux we might get some more insights.

 

[mike.jakubik]

Hello & Welcome to this FreeBSD community!

Unfortunately, I don't have a direct answer for you. However, I can share what I'd be doing in your position (which may or may not be helpful).

Have you tried increasing the transfer size -n option? 1 GB is not that much data over a 25Gbps link:


Have you tried to increase the size of the read/write buffer -l option?

Hi,

These options don't make any difference, it still peaks at around 11 down to 4, for an avg of 6-8. I have also tried copying a large file via ssh, I was only able to get around 300MB/s. This is stable so no debug options. I did share the output of my Linux run (you can see it in the attached screenshots), with Linux I get a steady 14.6Gb's and 0 retries. With FreeBSD it just seems to be all over the place, constantly changing, like the network stack doesn't know what to do with itself.

Thanks.

 

[SirDice]

with a fresh install of stable/13-ebea872f8

Can you post the output from uname -a because I can't tell at what time this 13-STABLE image was made.

 

[mike.jakubik]

Can you post the output from uname -a because I can't tell at what time this 13-STABLE image was made.

Sure, it is slightly newer now, but still the same results.

[root@db-01 ~]# uname -a
FreeBSD db-01.swift-api.com 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-40bbef56c: Thu Jun 9 14:43:26 EDT 2022 root@db-01.swift-api.com:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

 

[SirDice]

it is slightly newer now

Ok, good. I see a lot of commits from earlier this year, just wanted to make sure it's a recent 13-STABLE.

src - FreeBSD source tree

cgit.freebsd.org

 

[Max212]

maybe this could help?

iperf3 at 40Gbps and above

Note: the new version of iperf3 is multi-threaded, and makes this page mostly obsolete! Achieving line rate on a 40G or5

fasterdata.es.net

Did you try to use UDP protocol to to saturate bandwidth?

 

[mike.jakubik]

maybe this could help?

iperf3 at 40Gbps and above

Note: the new version of iperf3 is multi-threaded, and makes this page mostly obsolete! Achieving line rate on a 40G or5

fasterdata.es.net

Did you try to use UDP protocol to to saturate bandwidth?

This gives me a total of about 23Gb of traffic. I'm still curious though why a single test yields such low and sporadic results though, especially compared to Linux.

[root@db-02 ~]# s1:  Connecting to host db-01, port 5101
s3:  Connecting to host db-01, port 5103
s2:  Connecting to host db-01, port 5102
s3:  [  5] local 192.168.10.31 port 28885 connected to 192.168.10.30 port 5103
s1:  [  5] local 192.168.10.31 port 28883 connected to 192.168.10.30 port 5101
s2:  [  5] local 192.168.10.31 port 28886 connected to 192.168.10.30 port 5102
s1:  [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
s1:  [  5]   0.00-1.00   sec   827 MBytes  6.94 Gbits/sec    0   1.32 MBytes
s3:  [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
s3:  [  5]   0.00-1.00   sec   776 MBytes  6.51 Gbits/sec   87   1.25 MBytes
s2:  [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
s2:  [  5]   0.00-1.00   sec   814 MBytes  6.83 Gbits/sec    0   1.32 MBytes
s1:  [  5]   1.00-2.00   sec   861 MBytes  7.23 Gbits/sec    0   1.42 MBytes
s3:  [  5]   1.00-2.00   sec   805 MBytes  6.75 Gbits/sec    0   1.41 MBytes
s2:  [  5]   1.00-2.00   sec   849 MBytes  7.12 Gbits/sec    0   1.43 MBytes
s1:  [  5]   2.00-3.00   sec   924 MBytes  7.75 Gbits/sec    0   1.60 MBytes
s3:  [  5]   2.00-3.00   sec   939 MBytes  7.87 Gbits/sec    0   1.60 MBytes
s2:  [  5]   2.00-3.00   sec   927 MBytes  7.77 Gbits/sec    0   1.61 MBytes
s1:  [  5]   3.00-4.00   sec   938 MBytes  7.87 Gbits/sec    0   1.60 MBytes
s3:  [  5]   3.00-4.00   sec   934 MBytes  7.84 Gbits/sec    0   1.60 MBytes
s2:  [  5]   3.00-4.00   sec   934 MBytes  7.83 Gbits/sec    0   1.61 MBytes
s1:  [  5]   4.00-5.00   sec  1.08 GBytes  9.26 Gbits/sec    0   1.60 MBytes
s3:  [  5]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.60 MBytes
s2:  [  5]   4.00-5.00   sec   438 MBytes  3.68 Gbits/sec    2    924 KBytes
s1:  [  5]   5.00-6.00   sec   935 MBytes  7.85 Gbits/sec    0   1.60 MBytes
s3:  [  5]   5.00-6.00   sec   938 MBytes  7.87 Gbits/sec    0   1.60 MBytes
s2:  [  5]   5.00-6.00   sec   931 MBytes  7.81 Gbits/sec    0   1.60 MBytes
s1:  [  5]   6.00-7.00   sec   937 MBytes  7.86 Gbits/sec    0   1.61 MBytes
s3:  [  5]   6.00-7.00   sec   939 MBytes  7.88 Gbits/sec    0   1.60 MBytes
s2:  [  5]   6.00-7.00   sec   929 MBytes  7.79 Gbits/sec    0   1.60 MBytes
s1:  [  5]   7.00-8.00   sec   937 MBytes  7.86 Gbits/sec    0   1.61 MBytes
s3:  [  5]   7.00-8.00   sec   940 MBytes  7.89 Gbits/sec    0   1.60 MBytes
s2:  [  5]   7.00-8.00   sec   929 MBytes  7.79 Gbits/sec    0   1.61 MBytes
s1:  [  5]   8.00-9.00   sec   937 MBytes  7.86 Gbits/sec    0   1.61 MBytes
s3:  [  5]   8.00-9.00   sec   940 MBytes  7.88 Gbits/sec    0   1.60 MBytes
s2:  [  5]   8.00-9.00   sec   928 MBytes  7.79 Gbits/sec    0   1.61 MBytes
s1:  [  5]   9.00-10.00  sec   933 MBytes  7.82 Gbits/sec    0   1.61 MBytes
s1:  - - - - - - - - - - - - - - - - - - - - - - - - -
s1:  [ ID] Interval           Transfer     Bitrate         Retr
s1:  [  5]   0.00-10.00  sec  9.11 GBytes  7.83 Gbits/sec    0             sender
s1:  [  5]   0.00-10.00  sec  9.11 GBytes  7.83 Gbits/sec                  receiver
s1:
s1:  iperf Done.
s2:  [  5]   9.00-10.00  sec   932 MBytes  7.82 Gbits/sec    0   1.61 MBytes
s2:  - - - - - - - - - - - - - - - - - - - - - - - - -
s2:  [ ID] Interval           Transfer     Bitrate         Retr
s2:  [  5]   0.00-10.00  sec  8.41 GBytes  7.22 Gbits/sec    2             sender
s2:  [  5]   0.00-10.00  sec  8.41 GBytes  7.22 Gbits/sec                  receiver
s2:
s2:  iperf Done.
s3:  [  5]   9.00-10.00  sec   939 MBytes  7.88 Gbits/sec    0   1.60 MBytes
s3:  - - - - - - - - - - - - - - - - - - - - - - - - -
s3:  [ ID] Interval           Transfer     Bitrate         Retr
s3:  [  5]   0.00-10.00  sec  9.05 GBytes  7.78 Gbits/sec   87             sender
s3:  [  5]   0.00-10.00  sec  9.05 GBytes  7.78 Gbits/sec                  receiver
s3:
s3:  iperf Done.

 

[mike.jakubik]

I had a similar situation in the past (although with a 10 Gbps system). Just piping a huge file through the connection instead of using iperf3(1) yielded different results.

How exactly did you pipe it?