Single thread download over FreeBSD router - slow speed.

Hello everyone.
On the router (gateway) work FreeBSD with 10G network card.
And the problem is in single thread download over router, but in multi download speed increases so much.
Typicaly scheem:
DOWNLOAD FROMDOWNLOAD TOSPEED, Mbyte/sTHREAD (wget ftp://......)It's OK?
ClientRouter90+1Ok
RouterClient90+1Ok
ServerRouter180+1Ok
RouterServer180+1Ok
ClientServer10..201not OK, slow
ServerClient10..201not OK, slow
ClientServer80+manyOk
ServerClient80+manyOk
Client -> Router: 90Mbyte/s (OK)
Router->Client : 90Mbyte/s (OK)
Server -> Router: 180Mbyte/s (OK)
Router-> Server: 180Mbyte/s (OK)
Client ->Server (over router) : 10-20Mbyte/s (not OK)
Server->Client (over router) : 10-20Mbyte/s (not OK)
Comment:
When I run 1 ftp download from "CLIENT" to "ROUTER" I have speed over 90-95Mbyte (limiting by 1gb switch) - it's OK.
Some result from ROUTER to CLIENT ( 90+ Mbyte).
When I download from remote "SERVER" to "ROUTER" I have speed 180-200Mbyte (limiting by hdd?) - it's very OK.
Some result from "ROUTER" to remote "SERVER" (180-200Mbyte).
But when I run 1 ftp download from remote "SERVER" to "CLIENT" I have only 10..15..20Mbyte (In peak 23Mbyte). Some resurl in backwards download.
And whet I run miltiple download from "remote "SERVER" to "CLIENT" speed up to 80+Mbyte it's OK.
And some info about systems:
Code:
 2X CPU: Intel(R) Xeon(R) CPU E5-2689 0 @ 2.60GHz (2593.56-MHz K8-class CPU)
Code:
any of this sysctl does not matter:
net.link.ifqmaxlen=8192
net.isr.defaultqlimi=4096
net.isr.numthreads=4
hw.sfxge.tx_dpl_get_non_tcp_max=10240
net.graph.recvspace=128000
net.graph.maxdgram=64000
net.inet.ip.fw.one_pass=0
net.inet.icmp.icmplim=1500
kern.ipc.nmbclusters=65536

net.inet.flowtable.enable=0
sysctl net.bpf.maxinsns=1024
sysctl net.bpf.maxbufsize=1024000
sysctl net.bpf.bufsize=8192


net.inet.tcp.rfc1323=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendspace=1048576
net.inet.ip.intr_queue_maxlen=1000
net.inet.ip.dummynet.hash_size=256
kern.ipc.maxsockbuf=900000000
net.inet.tcp.sendspace=300000000
net.inet.tcp.recvspace=300000000
net.inet.udp.recvspace=300000000
net.inet.tcp.recvspace=1048576

Some IPFW rules, but from/to CLIENT and SERVER rules are first
Any CPU core not load over 50%.
Always Free memory.
Average traffic 200Kpps.
Network card driver sfxge
BGP working with 150 000 entries in routing table
Version: FreeBSD 12.2-RELEASE r366954 GENERIC amd64


Where starting find problem?
 
Hello & Welcome to this FreeBSD community!

In general, with these sorts of things you'd be best off providing a lot more details - especially how you acquired those numbers (eg. how you made those measurements).

benchmarks/iperf3 is also a good utility for these sorts of endeavors.
 
Hello & Welcome to this FreeBSD community!

In general, with these sorts of things you'd be best of providing a lot more details - especially how you acquired those numbers (eg. how you made those measurements).

benchmarks/iperf3 is also a good utility for these sorts of endeavors.
Its so simply: wget [in router I also use --bind-address] ftp://.....bigfile.avi
As ftp server i use standart ftpd on SERVER and ROUTER, in CLIENT I use windows 7 build in ftp server.
new CLIENT2 (bhyve@ROUTER at some network as CLIENT freebsd 13.1):

iperf 3 (as server side)iperf 3 (as client side)Transfer @ 10secBitrateRetr
ROUTERSERVER2.96 GBytes
2.96 GBytes
2.54 Gbits/sec
2.54 Gbits/sec
1237
-
ROUTERCLIENT21013 MBytes
1013 MBytes
850 Mbits/sec
849 Mbits/sec
0
-
SERVERROUTER3.12 GBytes
2.84 GBytes
2.68 Gbits/sec
2.40 Gbits/sec
961
-
SERVERCLIENT21.10 GBytes
1.10 GBytes
942 Mbits/sec
942 Mbits/sec
215
-
CLIENT2ROUTER1.30 GBytes
1.02 GBytes
1.11 Gbits/sec
843 Mbits/sec
0
-
CLIENT2SERVER108 MBytes
108 MBytes
90.7 Mbits/sec
88.7 Mbits/sec
7406
-
and raw output (to many letters):
root@SERVER # iperf3 -c ROUTER
Connecting to host ROUTER, port 5201
[ 5] local SERVER port 42475 connected to ROUTER port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 263 MBytes 2.20 Gbits/sec 0 479 KBytes
[ 5] 1.00-2.00 sec 295 MBytes 2.47 Gbits/sec 16 360 KBytes
[ 5] 2.00-3.00 sec 344 MBytes 2.89 Gbits/sec 0 548 KBytes
[ 5] 3.00-4.00 sec 282 MBytes 2.37 Gbits/sec 67 337 KBytes
[ 5] 4.00-5.00 sec 326 MBytes 2.74 Gbits/sec 0 515 KBytes
[ 5] 5.00-6.00 sec 329 MBytes 2.76 Gbits/sec 28 347 KBytes
[ 5] 6.00-7.00 sec 322 MBytes 2.70 Gbits/sec 0 525 KBytes
[ 5] 7.00-8.00 sec 290 MBytes 2.43 Gbits/sec 0 645 KBytes
[ 5] 8.00-9.00 sec 314 MBytes 2.63 Gbits/sec 0 745 KBytes
[ 5] 9.00-10.00 sec 267 MBytes 2.24 Gbits/sec 102 499 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.96 GBytes 2.54 Gbits/sec 213 sender
[ 5] 0.00-10.00 sec 2.96 GBytes 2.54 Gbits/sec receiver
[root@CLIENT]# iperf3 -c ROUTER
Connecting to host ROUTER, port 5201
[ 5] local CLIENT port 63367 connected to ROUTER port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 98.4 MBytes 825 Mbits/sec 0 1.02 MBytes
[ 5] 1.00-2.00 sec 102 MBytes 856 Mbits/sec 0 1.02 MBytes
[ 5] 2.00-3.00 sec 101 MBytes 849 Mbits/sec 0 1.02 MBytes
[ 5] 3.00-4.00 sec 101 MBytes 852 Mbits/sec 0 1.02 MBytes
[ 5] 4.00-5.00 sec 99.1 MBytes 832 Mbits/sec 0 1.02 MBytes
[ 5] 5.00-6.00 sec 102 MBytes 858 Mbits/sec 0 1.02 MBytes
[ 5] 6.00-7.00 sec 105 MBytes 880 Mbits/sec 0 1.02 MBytes
[ 5] 7.00-8.00 sec 104 MBytes 869 Mbits/sec 0 1.02 MBytes
[ 5] 8.00-9.00 sec 98.8 MBytes 828 Mbits/sec 0 1.02 MBytes
[ 5] 9.00-10.00 sec 101 MBytes 850 Mbits/sec 0 1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1013 MBytes 850 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1013 MBytes 849 Mbits/sec receiver

[root@ROUTER]# iperf3 -c SERVER
Connecting to host SERVER, port 5201
[ 5] local ROUTER port 16751 connected to SERVER port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 553 MBytes 4.64 Gbits/sec 172 273 KBytes
[ 5] 1.00-2.00 sec 290 MBytes 2.43 Gbits/sec 115 369 KBytes
[ 5] 2.00-3.00 sec 301 MBytes 2.52 Gbits/sec 66 454 KBytes
[ 5] 3.00-4.00 sec 291 MBytes 2.44 Gbits/sec 70 223 KBytes
[ 5] 4.00-5.00 sec 297 MBytes 2.49 Gbits/sec 107 217 KBytes
[ 5] 5.00-6.00 sec 297 MBytes 2.50 Gbits/sec 88 337 KBytes
[ 5] 6.00-7.00 sec 283 MBytes 2.37 Gbits/sec 76 355 KBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.34 Gbits/sec 123 254 KBytes
[ 5] 8.00-9.00 sec 299 MBytes 2.51 Gbits/sec 100 271 KBytes
[ 5] 9.00-10.00 sec 300 MBytes 2.52 Gbits/sec 44 296 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.12 GBytes 2.68 Gbits/sec 961 sender
[ 5] 0.00-10.17 sec 2.84 GBytes 2.40 Gbits/sec receiver

[root@CLIENT] # iperf3 -c SERVER
Connecting to host SERVER, port 5201
[ 5] local CLIENT port 43883 connected to SERVER port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 110 MBytes 919 Mbits/sec 14 345 KBytes
[ 5] 1.00-2.01 sec 115 MBytes 962 Mbits/sec 14 238 KBytes
[ 5] 2.01-3.00 sec 106 MBytes 891 Mbits/sec 15 375 KBytes
[ 5] 3.00-4.00 sec 118 MBytes 985 Mbits/sec 10 191 KBytes
[ 5] 4.00-5.00 sec 114 MBytes 956 Mbits/sec 0 606 KBytes
[ 5] 5.00-6.00 sec 114 MBytes 953 Mbits/sec 24 256 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 53 306 KBytes
[ 5] 7.00-8.00 sec 115 MBytes 963 Mbits/sec 14 404 KBytes
[ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 19 503 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 939 Mbits/sec 52 351 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 215 sender
[ 5] 0.00-10.01 sec 1.10 GBytes 942 Mbits/sec receiver

[root@router] # iperf3 -c CLIENT
Connecting to host CLIENT, port 5201
[ 5] local ROUTER port 27289 connected to CLIENT port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 380 MBytes 3.19 Gbits/sec 0 1.61 MBytes
[ 5] 1.00-2.00 sec 100 MBytes 840 Mbits/sec 0 1.61 MBytes
[ 5] 2.00-3.00 sec 90.8 MBytes 762 Mbits/sec 0 1.61 MBytes
[ 5] 3.00-4.00 sec 96.8 MBytes 812 Mbits/sec 0 1.61 MBytes
[ 5] 4.00-5.00 sec 102 MBytes 857 Mbits/sec 0 1.61 MBytes
[ 5] 5.00-6.00 sec 113 MBytes 942 Mbits/sec 0 1.61 MBytes
[ 5] 6.00-7.00 sec 105 MBytes 882 Mbits/sec 0 1.61 MBytes
[ 5] 7.00-8.00 sec 104 MBytes 874 Mbits/sec 0 1.61 MBytes
[ 5] 8.00-9.00 sec 116 MBytes 973 Mbits/sec 0 1.61 MBytes
[ 5] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.30 GBytes 1.11 Gbits/sec 0 sender
[ 5] 0.00-10.37 sec 1.02 GBytes 843 Mbits/sec receiver
[root@SERVER]# iperf3 -c CLIENT
Connecting to host CLIENT, port 5201
[ 5] local SERVER port 21913 connected to CLIENT port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 13.3 MBytes 111 Mbits/sec 881 2.85 KBytes
[ 5] 1.00-2.00 sec 9.87 MBytes 82.8 Mbits/sec 478 14.2 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.9 Mbits/sec 676 14.2 KBytes
[ 5] 3.00-4.00 sec 10.6 MBytes 88.5 Mbits/sec 870 11.5 KBytes
[ 5] 4.00-5.00 sec 10.6 MBytes 89.1 Mbits/sec 609 12.8 KBytes
[ 5] 5.00-6.00 sec 10.9 MBytes 91.5 Mbits/sec 765 1.43 KBytes
[ 5] 6.00-7.00 sec 11.0 MBytes 91.9 Mbits/sec 852 2.85 KBytes
[ 5] 7.00-8.00 sec 10.9 MBytes 91.7 Mbits/sec 733 8.59 KBytes
[ 5] 8.00-9.00 sec 10.3 MBytes 86.3 Mbits/sec 662 79.8 KBytes
[ 5] 9.00-10.00 sec 10.4 MBytes 87.5 Mbits/sec 880 9.91 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 108 MBytes 90.7 Mbits/sec 7406 sender
[ 5] 0.00-10.20 sec 108 MBytes 88.7 Mbits/sec receiver
But:
Code:
 [root@CLIENT] #wget ftp://SERVER.......   will take about 100Mbits/sec (as start peak at 150-180, and quikly decrease to 100)
 
Besides not having any security patches installed, keep in mind that 12.2-RELEASE is EoL since March 2022.
Ohc... I know it. So to short life time of the 12.x
I can try 13.1. I'll do it tonight next weekend, becouse system in production.
 
So to short life time of the 12.x
major versions have a minimum of 5 year support. But only the last minor version of a major branch is supported. FreeBSD 12 is supported until June 2024.
 
I found iperf for windows:
This host limited to 1gb - 150-200mbit/s to other client = 800-850mbit/s max.

In single-download over iperf: 200mbit/s (its so slow)
In multi-download over iperf in 10 streams : 836mbit/s (its wery well)
C:\iperf3.exe -c SERVER
Connecting to host SERVER, port 5201
[ 4] local CLIENT port 62125 connected to SERVER port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 25.4 MBytes 213 Mbits/sec
[ 4] 1.00-2.00 sec 27.5 MBytes 231 Mbits/sec
[ 4] 2.00-3.00 sec 26.6 MBytes 223 Mbits/sec
[ 4] 3.00-4.00 sec 27.2 MBytes 229 Mbits/sec
[ 4] 4.00-5.00 sec 28.4 MBytes 238 Mbits/sec
[ 4] 5.00-6.00 sec 28.0 MBytes 235 Mbits/sec
[ 4] 6.00-7.00 sec 20.2 MBytes 170 Mbits/sec
[ 4] 7.00-8.00 sec 28.4 MBytes 238 Mbits/sec
[ 4] 8.00-9.00 sec 28.9 MBytes 242 Mbits/sec
[ 4] 9.00-10.00 sec 18.6 MBytes 156 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 259 MBytes 217 Mbits/sec sender
[ 4] 0.00-10.00 sec 259 MBytes 217 Mbits/sec receiver

iperf Done.
C:\iperf3.exe -c ROUTER
Connecting to host ROUTER, port 5201
[ 4] local CLIENT port 62158 connected to ROUTER port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 79.5 MBytes 667 Mbits/sec
[ 4] 1.00-2.00 sec 99.4 MBytes 834 Mbits/sec
[ 4] 2.00-3.00 sec 97.8 MBytes 820 Mbits/sec
[ 4] 3.00-4.00 sec 100 MBytes 839 Mbits/sec
[ 4] 4.00-5.00 sec 100 MBytes 839 Mbits/sec
[ 4] 5.00-6.00 sec 99.2 MBytes 833 Mbits/sec
[ 4] 6.00-7.00 sec 99.8 MBytes 837 Mbits/sec
[ 4] 7.00-8.00 sec 99.9 MBytes 838 Mbits/sec
[ 4] 8.00-9.00 sec 99.5 MBytes 835 Mbits/sec
[ 4] 9.00-10.00 sec 99.9 MBytes 838 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 975 MBytes 818 Mbits/sec sender
[ 4] 0.00-10.00 sec 975 MBytes 818 Mbits/sec receiver

iperf Done.


iperf -c SERVER -P 10
skipp some text.....

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 83.1 MBytes 69.7 Mbits/sec sender
[ 4] 0.00-10.00 sec 82.9 MBytes 69.6 Mbits/sec receiver
[ 6] 0.00-10.00 sec 102 MBytes 85.7 Mbits/sec sender
[ 6] 0.00-10.00 sec 102 MBytes 85.6 Mbits/sec receiver
[ 8] 0.00-10.00 sec 85.1 MBytes 71.4 Mbits/sec sender
[ 8] 0.00-10.00 sec 84.9 MBytes 71.2 Mbits/sec receiver
[ 10] 0.00-10.00 sec 105 MBytes 88.2 Mbits/sec sender
[ 10] 0.00-10.00 sec 105 MBytes 88.1 Mbits/sec receiver
[ 12] 0.00-10.00 sec 109 MBytes 91.4 Mbits/sec sender
[ 12] 0.00-10.00 sec 109 MBytes 91.3 Mbits/sec receiver
[ 14] 0.00-10.00 sec 86.8 MBytes 72.8 Mbits/sec sender
[ 14] 0.00-10.00 sec 86.6 MBytes 72.7 Mbits/sec receiver
[ 16] 0.00-10.00 sec 109 MBytes 91.5 Mbits/sec sender
[ 16] 0.00-10.00 sec 109 MBytes 91.5 Mbits/sec receiver
[ 18] 0.00-10.00 sec 98.6 MBytes 82.7 Mbits/sec sender
[ 18] 0.00-10.00 sec 98.5 MBytes 82.6 Mbits/sec receiver
[ 20] 0.00-10.00 sec 111 MBytes 92.9 Mbits/sec sender
[ 20] 0.00-10.00 sec 111 MBytes 92.8 Mbits/sec receiver
[ 22] 0.00-10.00 sec 106 MBytes 89.3 Mbits/sec sender
[ 22] 0.00-10.00 sec 106 MBytes 89.2 Mbits/sec receiver
[SUM] 0.00-10.00 sec 996 MBytes 836 Mbits/sec sender
[SUM] 0.00-10.00 sec 995 MBytes 835 Mbits/sec receiver
 
Back
Top