The first set of numbers are very good, you have single-digit milliseconds jitter. And the second set is not terrible either; being accurate to withim 130ms is pretty good. Clearly, the correct fix is to make the network not overload.
Yes, and that is the difficult part. 130ms is rather bad - there is some spec that VoIP demands 40ms, and in any case trying to read tcpdumps across the cloud is impractical that way.
I tried a lot of things to make this better, I'm running ntpd and all routing components in rtprio, I try to give bandwidth priviledge to ntp, and nothing did lead to a significant success.
Until I figured where the problem actually is: the congestion algorithms (reno, cdg) are not suitable. They feed packets into the link for as long as the tcp window does allow (and the tcp window does not care about the link bandwidth, only about the buffers at the endpoints), until all the buffers on the way are full. Then packets get dropped, and only then the congestion algorithm will react: it will limit the traffic, the packet loss goes away, and then all starts anew.
The outcome: half of the time the link bandwidth is not fully utilized, packet loss is high nevertheless, and the roundtrip delay fluctuates heavily and creates jitter.
Finally I figured how vegas works - it works completely different and aims to keep the delay constant - and now it looks like this:
Code:
remote refid st t when poll reach delay offset jitter
==============================================================================
+ntp3.linocomm.n 130.149.17.21 2 u 477 1024 377 33.635 +0.642 5.954
*stratum2-4.NTP. 129.70.137.82 2 u 959 1024 377 47.236 +1.827 2.566
+2a01:4f8:141:28 131.188.3.223 2 u 467 1024 377 37.507 +2.310 1.734
+mail.rleh.de 130.149.17.8 2 u 971 1024 377 33.672 +1.545 2.737
#daphne.ea1145.o 131.188.3.223 2 u 151 1024 377 33.584 +1.635 1.635
+2001:41d0:700:1 131.188.3.222 2 u 648 1024 377 30.743 +1.408 1.404
-srv.hueske-edv. 36.224.68.195 2 u 702 1024 377 39.281 +3.483 2.685
#ns2.madavi.de 131.188.3.220 2 u 555 1024 377 39.299 +2.870 4.666
What is obvious here: the absolute delay is 10ms more. This is because I have kern.hz=200 and net.inet.ip.dummynet.io_fast=1. The latter means that the dummynet pipes and queues will only engage when the link is actually filled, otherwise packets bypass dummynet and are sent directly. And dummynet runs on the HZ clock. So now dummynet is always engaged, the link is always fully saturated, packet loss is below what TCP SACK can handle inline, and the jitter is gone.
I think this is how it was intended to work.
Code:
net.inet.tcp.cc.algorithm="vegas"
net.inet.tcp.cc.vegas.beta="14"
net.inet.tcp.cc.vegas.alpha="8"
This works with nested VPNs, so the alpha and beta need to be larger than the default. [Alpha and Beta are the min and max of surplus packets inflight to fill the buffers on the way. Vegas measures the natural -minimum- roundtrip time, and computes the number of packets that should be naturally inflight at the current sending rate. If the actual number is above this+beta, the rate is reduced, if it is below this+alpha, the rate is increased.]
The main problem with vegas is that it is fairness-based and therefore cannot compete with the other more aggressive (and less effective) algorithms on the same link. As it is so often in life.
