Hello everyone.
On the router (gateway) work FreeBSD with 10G network card.
And the problem is in single thread download over router, but in multi download speed increases so much.
Typicaly scheem:
Some IPFW rules, but from/to CLIENT and SERVER rules are first
Any CPU core not load over 50%.
Always Free memory.
Average traffic 200Kpps.
Network card driver sfxge
BGP working with 150 000 entries in routing table
Version: FreeBSD 12.2-RELEASE r366954 GENERIC amd64
Where starting find problem?
On the router (gateway) work FreeBSD with 10G network card.
And the problem is in single thread download over router, but in multi download speed increases so much.
Typicaly scheem:
DOWNLOAD FROM | DOWNLOAD TO | SPEED, Mbyte/s | THREAD (wget ftp://......) | It's OK? |
Client | Router | 90+ | 1 | Ok |
Router | Client | 90+ | 1 | Ok |
Server | Router | 180+ | 1 | Ok |
Router | Server | 180+ | 1 | Ok |
Client | Server | 10..20 | 1 | not OK, slow |
Server | Client | 10..20 | 1 | not OK, slow |
Client | Server | 80+ | many | Ok |
Server | Client | 80+ | many | Ok |
And some info about systems:Client -> Router: 90Mbyte/s (OK)
Router->Client : 90Mbyte/s (OK)
Server -> Router: 180Mbyte/s (OK)
Router-> Server: 180Mbyte/s (OK)
Client ->Server (over router) : 10-20Mbyte/s (not OK)
Server->Client (over router) : 10-20Mbyte/s (not OK)
Comment:
When I run 1 ftp download from "CLIENT" to "ROUTER" I have speed over 90-95Mbyte (limiting by 1gb switch) - it's OK.
Some result from ROUTER to CLIENT ( 90+ Mbyte).
When I download from remote "SERVER" to "ROUTER" I have speed 180-200Mbyte (limiting by hdd?) - it's very OK.
Some result from "ROUTER" to remote "SERVER" (180-200Mbyte).
But when I run 1 ftp download from remote "SERVER" to "CLIENT" I have only 10..15..20Mbyte (In peak 23Mbyte). Some resurl in backwards download.
And whet I run miltiple download from "remote "SERVER" to "CLIENT" speed up to 80+Mbyte it's OK.
Code:
2X CPU: Intel(R) Xeon(R) CPU E5-2689 0 @ 2.60GHz (2593.56-MHz K8-class CPU)
Code:
any of this sysctl does not matter:
net.link.ifqmaxlen=8192
net.isr.defaultqlimi=4096
net.isr.numthreads=4
hw.sfxge.tx_dpl_get_non_tcp_max=10240
net.graph.recvspace=128000
net.graph.maxdgram=64000
net.inet.ip.fw.one_pass=0
net.inet.icmp.icmplim=1500
kern.ipc.nmbclusters=65536
net.inet.flowtable.enable=0
sysctl net.bpf.maxinsns=1024
sysctl net.bpf.maxbufsize=1024000
sysctl net.bpf.bufsize=8192
net.inet.tcp.rfc1323=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendspace=1048576
net.inet.ip.intr_queue_maxlen=1000
net.inet.ip.dummynet.hash_size=256
kern.ipc.maxsockbuf=900000000
net.inet.tcp.sendspace=300000000
net.inet.tcp.recvspace=300000000
net.inet.udp.recvspace=300000000
net.inet.tcp.recvspace=1048576
Some IPFW rules, but from/to CLIENT and SERVER rules are first
Any CPU core not load over 50%.
Always Free memory.
Average traffic 200Kpps.
Network card driver sfxge
BGP working with 150 000 entries in routing table
Version: FreeBSD 12.2-RELEASE r366954 GENERIC amd64
Where starting find problem?