how to hose TCP to use only half the available bandwidth

PMc · Feb 23, 2023

I have an application that transfers data at only half of what would be available bandwidth (or 2/3 when two threads are sending).

But the connection is TCP, and I thought the kernel would do the flow-control there. So I put the blame on my mess of nested tunnels and the generally narrow link (~2 Mbit/sec WAN).

Now I fixed that mess, i.e. put traffic shapers in and see what can work, thereby bringing ping on full saturation down from 250 ms to 60 ms, and now ssh and https work fine. I tested with even weirder rabbithole constructs, and they run at full saturation. Only that application doesn't.

Code:

09:31:45.871324 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2143354:2144673, ack 1, win 1040, options [nop,nop,TS val 1473408532 ecr 3301313561], length 1319
09:31:45.916328 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2144673:2145992, ack 1, win 1040, options [nop,nop,TS val 1473408587 ecr 3301313607], length 1319
09:31:45.926327 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2145992:2147311, ack 1, win 1040, options [nop,nop,TS val 1473408587 ecr 3301313607], length 1319
09:31:45.965843 IP 192.168.99.1.9103 > 192.168.97.17.47590: Flags [.], ack 2144673, win 1040, options [nop,nop,TS val 3301313846 ecr 1473408532], length 0
09:31:45.977413 IP 192.168.99.1.9103 > 192.168.97.17.47590: Flags [.], ack 2147311, win 1020, options [nop,nop,TS val 3301313856 ecr 1473408587], length 0
09:31:46.171359 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2147311:2148630, ack 1, win 1040, options [nop,nop,TS val 1473408817 ecr 3301313846], length 1319
09:31:46.176338 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2148630:2149949, ack 1, win 1040, options [nop,nop,TS val 1473408817 ecr 3301313846], length 1319
09:31:46.186358 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2149949:2151268, ack 1, win 1040, options [nop,nop,TS val 1473408827 ecr 3301313856], length 1319
09:31:46.191330 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2151268:2152587, ack 1, win 1040, options [nop,nop,TS val 1473408827 ecr 3301313856], length 1319
09:31:46.228385 IP 192.168.99.1.9103 > 192.168.97.17.47590: Flags [.], ack 2149949, win 1020, options [nop,nop,TS val 3301314108 ecr 1473408817], length 0
09:31:46.254554 IP 192.168.99.1.9103 > 192.168.97.17.47590: Flags [.], ack 2152587, win 1020, options [nop,nop,TS val 3301314123 ecr 1473408827], length 0
>>> what is it waiting for ??? <<<
09:31:46.431325 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2152587:2153906, ack 1, win 1040, options [nop,nop,TS val 1473409102 ecr 3301314123], length 1319
09:31:46.436330 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2153906:2155225, ack 1, win 1040, options [nop,nop,TS val 1473409102 ecr 3301314123], length 1319
09:31:46.446334 IP 192.168.97.17.47590 > 192.168.99.1.9103: Flags [.], seq 2155225:2156544, ack 1, win 1040, options [nop,nop,TS val 1473409102 ecr 3301314123], length 1319

Code:

8.450039607 0.020239260 poll({ 22/POLLOUT|POLLWRBAND },1,10000) = 1 (0x1)
8.450454142 0.000065407 write(22,"\^W\^C\^C\^C\M^U\M-38\M-&mhJ\240"...,922) = 922 (0x39a)
8.450572234 0.000028607 fcntl(22,F_SETFL,O_RDWR) = 0 (0x0)
8.450707784 0.000055497 read(6,"\^A\M-Q\^A\0\^A\0\0\0\0\0G\M-lk"...,65536) = 65536 (0x10000)
8.463065391 0.000029543 fcntl(22,F_GETFL,)       = 2 (0x2)
8.463198288 0.000034799 fcntl(22,F_SETFL,O_RDWR|O_NONBLOCK) = 0 (0x0)
8.463620081 0.000064920 write(22,"\^W\^C\^C@\^Qr\M-N\M^Z\M^P6D\M-S"...,16406) ERR#35 'Resource temporarily unavailable'
8.472786898 0.009032704 poll({ 22/POLLOUT|POLLWRBAND },1,10000) = 1 (0x1)
8.473162083 0.000068892 write(22,"\^W\^C\^C@\^Qr\M-N\M^Z\M^P6D\M-S"...,16406) = 4576 (0x11e0)
8.473449626 0.000027335 write(22,"\M-+UFcR\M^Y7~\n;\M-E\M^^\M-;0"...,11830) ERR#35 'Resource temporarily unavailable'
8.480161835 0.006614703 poll({ 22/POLLOUT|POLLWRBAND },1,10000) = 1 (0x1)
8.480501420 0.000054956 write(22,"\M-+UFcR\M^Y7~\n;\M-E\M^^\M-;0"...,11830) = 2638 (0xa4e)
8.480879519 0.000060052 write(22,"\M-2\M-k\M-d\^C\M-Vt\M-t\M^[\M^\"...,9192) ERR#35 'Resource temporarily unavailable'
8.488571034 0.007558037 poll({ 22/POLLOUT|POLLWRBAND },1,10000) = 1 (0x1)

Doesn't look obviousely wrong...

covacat · Feb 23, 2023

does changing net.inet.tcp.delayed_ack to 0 make any difference ?

PMc · Feb 23, 2023

ssh does too many things to understand the truss output, the webserver is rather difficult to tackle, but there is netcat and the google way...
So, at least now I have a log for which syscalls are minimally needed to do it properly.

But, damn, after then restarting the server side, the problem isn't visible anymore.

covacat said:
does changing net.inet.tcp.delayed_ack to 0 make any difference ?

Thank you, noted.

I'd rather prefer to fix that application, because I know they tend to do the things on their own, and they usually do them wrongly.

PMc · Feb 23, 2023

covacat said:
does changing net.inet.tcp.delayed_ack to 0 make any difference ?

Apparently no - maybe it applies only to new sockets?

But I think I see what's happening. or at least part of it:

It starts with some 5kB window.
It then grows to 30kB, and things do still work.
With a bw of 2Mbit and a looptime of 60-75 ms, this would already be more than enough.

But then it increases further to 60kB, and now the bufferbloat happens. The looptime is now at 300ms, until something on the way does droptail (this is intended) and throws away half of the crap.

Then big discussion, lots of SACK, and it starts over again.

Questions:
- what do the other programs differently?
- how best to get rid of this?

Concerning ssh: the same thing is happening, but it's handled smoothly and doesn't create a problem:

Code:

21:20:37.708202 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 523644, win 1560, options [nop,nop,TS val 697802339 ecr 3639761867,nop,nop,sack 1 {524963:565852}], length 0
21:20:37.708233 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 582999:584318, ack 144, win 1040, options [nop,nop,TS val 3639762707 ecr 697802339], length 1319
21:20:37.710884 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 565852, win 901, options [nop,nop,TS val 697802385 ecr 3639762327], length 0
21:20:37.710920 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 584318:585637, ack 144, win 1040, options [nop,nop,TS val 3639762707 ecr 697802385], length 1319
21:20:37.711178 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 565852, win 1560, options [nop,nop,TS val 697802385 ecr 3639762327], length 0
21:20:37.711208 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 585637:586956, ack 144, win 1040, options [nop,nop,TS val 3639762707 ecr 697802385], length 1319
21:20:37.733562 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 571128, win 1540, options [nop,nop,TS val 697802406 ecr 3639762387], length 0
21:20:37.733618 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 586956:588275, ack 144, win 1040, options [nop,nop,TS val 3639762732 ecr 697802406], length 1319
21:20:37.733622 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 588275:589594, ack 144, win 1040, options [nop,nop,TS val 3639762732 ecr 697802406], length 1319
21:20:37.733624 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 589594:590913, ack 144, win 1040, options [nop,nop,TS val 3639762732 ecr 697802406], length 1319
21:20:37.733626 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 590913:592232, ack 144, win 1040, options [nop,nop,TS val 3639762732 ecr 697802406], length 1319
21:20:37.747355 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 573766, win 1540, options [nop,nop,TS val 697802426 ecr 3639762442], length 0
...
21:20:42.543328 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1104004, win 1540, options [nop,nop,TS val 697807224 ecr 3639767172], length 0
21:20:42.543370 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1143574:1144893, ack 288, win 1040, options [nop,nop,TS val 3639767542 ecr 697807224], length 1319
21:20:42.543377 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1144893:1146212, ack 288, win 1040, options [nop,nop,TS val 3639767542 ecr 697807224], length 1319
21:20:42.543379 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1146212:1147531, ack 288, win 1040, options [nop,nop,TS val 3639767542 ecr 697807224], length 1319
21:20:42.562807 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1106642, win 1540, options [nop,nop,TS val 697807241 ecr 3639767172], length 0
21:20:42.562844 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1147531:1148850, ack 288, win 1040, options [nop,nop,TS val 3639767562 ecr 697807241], length 1319
21:20:42.562850 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1148850:1150169, ack 288, win 1040, options [nop,nop,TS val 3639767562 ecr 697807241], length 1319
21:20:42.605614 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1109280, win 1540, options [nop,nop,TS val 697807282 ecr 3639767202], length 0
21:20:42.605659 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1150169:1151488, ack 288, win 1040, options [nop,nop,TS val 3639767602 ecr 697807282], length 1319
21:20:42.605666 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1151488:1152807, ack 288, win 1040, options [nop,nop,TS val 3639767602 ecr 697807282], length 1319
21:20:42.643353 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1110599, win 1560, options [nop,nop,TS val 697807319 ecr 3639767227,nop,nop,sack 1 {1113237:1114556}], length 0
21:20:42.643386 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1152807:1154126, ack 288, win 1040, options [nop,nop,TS val 3639767642 ecr 697807319], length 1319
21:20:42.643597 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1110599, win 1560, options [nop,nop,TS val 697807326 ecr 3639767227,nop,nop,sack 1 {1113237:1115875}], length 0
21:20:42.643623 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1154126:1155445, ack 288, win 1040, options [nop,nop,TS val 3639767642 ecr 697807326], length 1319
21:20:42.653164 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1110599, win 1560, options [nop,nop,TS val 697807335 ecr 3639767227,nop,nop,sack 1 {1113237:1117194}], length 0
21:20:42.653189 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1110599:1111918, ack 288, win 1040, options [nop,nop,TS val 3639767652 ecr 697807335], length 1319
...
21:20:43.064942 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1146212, win 1560, options [nop,nop,TS val 697807738 ecr 3639767697,nop,nop,sack 1 {1147531:1165997}], length 0
21:20:43.064967 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1184463:1185782, ack 288, win 1040, options [nop,nop,TS val 3639768062 ecr 697807738], length 1319
21:20:43.088315 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1146212, win 1560, options [nop,nop,TS val 697807770 ecr 3639767697,nop,nop,sack 1 {1147531:1167316}], length 0
21:20:43.088340 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1185782:1187101, ack 288, win 1040, options [nop,nop,TS val 3639768087 ecr 697807770], length 1319
21:20:43.100719 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1167316, win 1231, options [nop,nop,TS val 697807783 ecr 3639768002], length 0
21:20:43.100753 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1187101:1188420, ack 288, win 1040, options [nop,nop,TS val 3639768097 ecr 697807783], length 1319
21:20:43.100877 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [P.], seq 288:324, ack 1167316, win 1560, options [nop,nop,TS val 697807783 ecr 3639768002], length 36
21:20:43.119806 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1169954, win 1540, options [nop,nop,TS val 697807801 ecr 3639768002], length 0
21:20:43.119848 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1188420:1189739, ack 324, win 1040, options [nop,nop,TS val 3639768117 ecr 697807783], length 1319
21:20:43.119855 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1189739:1191058, ack 324, win 1040, options [nop,nop,TS val 3639768117 ecr 697807783], length 1319
21:20:43.119857 IP 192.168.97.17.62291 > 192.168.99.1.22: Flags [.], seq 1191058:1192377, ack 324, win 1040, options [nop,nop,TS val 3639768117 ecr 697807783], length 1319
21:20:43.153855 IP 192.168.99.1.22 > 192.168.97.17.62291: Flags [.], ack 1172592, win 1540, options [nop,nop,TS val 697807834 ecr 3639768002], length 0

To the contrast, the problematic application:

Code:

21:32:27.655339 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5297983, win 1020, options [nop,nop,TS val 1295540419 ecr 1182142750], length 0
21:32:27.655575 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5313811:5315130, ack 0, win 1040, options [nop,nop,TS val 1182142885 ecr 1295540419], length 1319
21:32:27.655580 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5315130:5316449, ack 0, win 1040, options [nop,nop,TS val 1182142885 ecr 1295540419], length 1319
21:32:27.655705 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5300621, win 1020, options [nop,nop,TS val 1295540419 ecr 1182142765], length 0
21:32:27.655777 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5316449:5317768, ack 0, win 1040, options [nop,nop,TS val 1182142885 ecr 1295540419], length 1319
21:32:27.655783 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5317768:5319087, ack 0, win 1040, options [nop,nop,TS val 1182142885 ecr 1295540419], length 1319
21:32:27.662551 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5303259, win 1020, options [nop,nop,TS val 1295540426 ecr 1182142785], length 0
21:32:27.671638 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5305897, win 1020, options [nop,nop,TS val 1295540442 ecr 1182142800], length 0
21:32:27.672416 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5319087:5320406, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672422 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5320406:5321725, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672425 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5321725:5323044, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672427 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5323044:5324363, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672429 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5324363:5325682, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672431 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5325682:5327001, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672433 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5327001:5328320, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672435 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5328320:5329639, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672437 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5329639:5330958, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672439 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5330958:5332277, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672441 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5332277:5333596, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672443 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5333596:5334915, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672445 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5334915:5336234, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672447 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5336234:5337553, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672449 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5337553:5338872, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672451 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5338872:5340191, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672453 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5340191:5341510, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672455 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5341510:5342829, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672457 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5342829:5344148, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672459 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5344148:5345467, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672461 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5345467:5346786, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672463 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5346786:5348105, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672465 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5348105:5349424, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672467 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5349424:5350743, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672469 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5350743:5352062, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672472 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5352062:5353381, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672474 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5353381:5354700, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672476 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5354700:5356019, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672478 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5356019:5357338, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672480 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5357338:5358657, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672482 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5358657:5359976, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672484 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5359976:5361295, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672486 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5361295:5362614, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672489 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5362614:5363933, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672491 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5363933:5365252, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672493 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5365252:5366571, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672495 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5366571:5367890, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672497 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5367890:5369209, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.672499 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5369209:5370528, ack 0, win 1040, options [nop,nop,TS val 1182142905 ecr 1295540442], length 1319
21:32:27.683974 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5308535, win 1020, options [nop,nop,TS val 1295540457 ecr 1182142855], length 0
21:32:27.684064 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5370528:5371847, ack 0, win 1040, options [nop,nop,TS val 1182142915 ecr 1295540457], length 1319
21:32:27.684078 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5371847:5373166, ack 0, win 1040, options [nop,nop,TS val 1182142915 ecr 1295540457], length 1319
...
21:32:28.016802 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5366571:5367890, ack 0, win 1040, options [nop,nop,TS val 1182143250 ecr 1295540774], length 1319
21:32:28.016808 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5367890:5369209, ack 0, win 1040, options [nop,nop,TS val 1182143250 ecr 1295540774], length 1319
21:32:28.016810 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5369209:5370528, ack 0, win 1040, options [nop,nop,TS val 1182143250 ecr 1295540781], length 1319
21:32:28.016813 IP 192.168.97.17.12071 > 192.168.99.1.9103: Flags [.], seq 5371847:5373166, ack 0, win 1040, options [nop,nop,TS val 1182143250 ecr 1295540781], length 1319
21:32:28.018534 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5349424, win 1040, options [nop,nop,TS val 1295540788 ecr 1182142905,nop,nop,sack 2 {5373166:5386356}{5370528:5371847}], length 0
21:32:28.028681 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5349424, win 1040, options [nop,nop,TS val 1295540797 ecr 1182142905,nop,nop,sack 2 {5373166:5387675}{5370528:5371847}], length 0
21:32:28.033803 IP 192.168.99.1.9103 > 192.168.97.17.12071: Flags [.], ack 5349424, win 1040, options [nop,nop,TS val 1295540803 ecr 1182142905,nop,nop,sack 2 {5373166:5388994}{5370528:5371847}], length 0

PMc · Feb 25, 2023

I finally read the application's docs and found this:

Maximum Network Buffer Size

Type:

PINT32

This directive specifies the initial network buffer size to
use. This size will be adjusted down if it is too large until it
is accepted by the OS. Please use care in setting this value since
if it is too large, it will be trimmed by 512 bytes until the OS
is happy, which may require a large number of system calls. The
default value is 65,536 bytes.

I checked the source where this variable is used, and found a comment explaining that this (the 64k byte part) is not done anymore: one does no longer set the buffer size by default, because the admin might have set a systemwide default that should be honored. The buffer size will only be set when this config variable is explicitely set.

So I set it to 27k, i.e. somewhere above my bwd product, and voila, the speed is significantly better.

But this is not a solution: whenever the application decides to run two or more threads over the same link, then every socket will use these 27k, and there will be buffer overruns again. Also, the handling of missing packets is still more than ugly:

Code:

23:43:41.274323 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9302321, win 1020, options [nop,nop,TS val 2783153227 ecr 3246660656], length 0
23:43:41.274425 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9326063:9327382, ack 0, win 1040, options [nop,nop,TS val 3246660856 ecr 2783153227], length 1319
23:43:41.274440 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9327382:9328701, ack 0, win 1040, options [nop,nop,TS val 3246660856 ecr 2783153227], length 1319
23:43:41.275076 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9304959, win 1020, options [nop,nop,TS val 2783153228 ecr 3246660676], length 0
23:43:41.275119 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9328701:9330020, ack 0, win 1040, options [nop,nop,TS val 3246660856 ecr 2783153228], length 1319
23:43:41.294825 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153244 ecr 3246660686,nop,nop,sack 1 {9307597:9308916}], length 0
23:43:41.294873 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9330020:9331339, ack 0, win 1040, options [nop,nop,TS val 3246660876 ecr 2783153244], length 1319
23:43:41.294888 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9331339:9332658, ack 0, win 1040, options [nop,nop,TS val 3246660876 ecr 2783153244], length 1319
23:43:41.294966 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153244 ecr 3246660686,nop,nop,sack 1 {9307597:9310235}], length 0
23:43:41.294987 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153244 ecr 3246660686,nop,nop,sack 1 {9307597:9311554}], length 0
23:43:41.295008 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9306278:9307597, ack 0, win 1040, options [nop,nop,TS val 3246660876 ecr 2783153244], length 1319
23:43:41.295064 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153244 ecr 3246660686,nop,nop,sack 1 {9307597:9312873}], length 0
23:43:41.299587 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153248 ecr 3246660686,nop,nop,sack 1 {9307597:9314192}], length 0
23:43:41.306421 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153255 ecr 3246660686,nop,nop,sack 1 {9307597:9315511}], length 0
23:43:41.316717 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153263 ecr 3246660686,nop,nop,sack 1 {9307597:9316830}], length 0
23:43:41.329368 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153269 ecr 3246660686,nop,nop,sack 1 {9307597:9318149}], length 0
23:43:41.329388 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153279 ecr 3246660686,nop,nop,sack 1 {9307597:9319468}], length 0
23:43:41.335952 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153288 ecr 3246660686,nop,nop,sack 1 {9307597:9320787}], length 0
23:43:41.345740 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153297 ecr 3246660686,nop,nop,sack 1 {9307597:9322106}], length 0
23:43:41.352161 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153303 ecr 3246660686,nop,nop,sack 1 {9307597:9323425}], length 0
23:43:41.359338 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153311 ecr 3246660686,nop,nop,sack 1 {9307597:9324744}], length 0
23:43:41.369044 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153318 ecr 3246660686,nop,nop,sack 1 {9307597:9326063}], length 0
23:43:41.377288 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153326 ecr 3246660686,nop,nop,sack 1 {9307597:9327382}], length 0
23:43:41.383728 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153332 ecr 3246660686,nop,nop,sack 1 {9307597:9328701}], length 0
23:43:41.388707 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153338 ecr 3246660686,nop,nop,sack 1 {9307597:9330020}], length 0
23:43:41.395589 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153348 ecr 3246660686,nop,nop,sack 1 {9307597:9331339}], length 0
23:43:41.405935 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9306278, win 1040, options [nop,nop,TS val 2783153357 ecr 3246660686,nop,nop,sack 1 {9307597:9332658}], length 0
23:43:41.414541 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9332658, win 628, options [nop,nop,TS val 2783153364 ecr 3246660876], length 0
23:43:41.414577 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9332658, win 1009, options [nop,nop,TS val 2783153365 ecr 3246660876], length 0
23:43:41.414586 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [P.], seq 9332658:9332902, ack 0, win 1040, options [nop,nop,TS val 3246660996 ecr 2783153364], length 244
23:43:41.414613 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9332902:9334221, ack 0, win 1040, options [nop,nop,TS val 3246660996 ecr 2783153365], length 1319
23:43:41.476673 IP 192.168.99.1.9103 > 192.168.97.17.41787: Flags [.], ack 9334221, win 1020, options [nop,nop,TS val 2783153420 ecr 3246660996], length 0
23:43:41.476718 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9334221:9335540, ack 0, win 1040, options [nop,nop,TS val 3246661056 ecr 2783153420], length 1319
23:43:41.476722 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9335540:9336859, ack 0, win 1040, options [nop,nop,TS val 3246661056 ecr 2783153420], length 1319
23:43:41.476723 IP 192.168.97.17.41787 > 192.168.99.1.9103: Flags [.], seq 9336859:9338178, ack 0, win 1040, options [nop,nop,TS val 3246661056 ecr 2783153420], length 1319

One can see at the top that the configured 27k is fully used:
seq 9328701 - ack 9302321 = 26380
Then a packet is lost - and the transfer stops entirely. This is obvious, because the sender cannot remove the oldest packet from the buffer, as it needs to be resent. And since the buffer is fully used, there is no space free for further data.

But what I don't understand: when I read tcp(4), there are hundreds of options and features for automatic self-tuning. Most of them are already set to best defaults, so usually the network does work optimally out of the box, and only extreme infrastructure needs further adjusts (e.g 10xGbit, satlink, packet-radio).
And for netcat, ssh, webserver it obviousely works that way.

So, how can an application open a socket in a way that all these features are disengaged and we're like back in 1985 when we did our network buffer calculations manually?!

PMc · Feb 27, 2023

Netcat does:
socket(PF_INET,SOCK_STREAM,IPPROTO_TCP) = 3 (0x3)
The application does:
socket(PF_INET6,SOCK_STREAM,0) = 7 (0x7)

Might this make a difference?

(I tried to change the application to also set IPPROTO_TCP, but now the results are almost worse.)

how to hose TCP to use only half the available bandwidth

PMc

covacat

PMc

PMc

PMc

PMc