Solved how does bandwidth limit in UDP work?

[PMc]

From what we learn, TCP can and will adapt to the bandwidth by the window/ACK mechanism. So if there is a bandwidth limit, we only need to delay the packets, and TCP will automatically adjust to it.
But in UDP there is no such thing. So when the packets are delayed, they will pile up, and we cannot do anything else than throw them away. There is no way to tell the connection to slow down, because there is no notion of a connection.

So, when using a dummynet pipe to limit bandwidth, I keep getting dropped packets and such ugly messages:

openvpn[86893]: write UDP: No buffer space available (code=55)

And it seems there is nothing that can be done about.

But then, how is this done on normal network connections? There are fast ethernet connections and slow serial ones, how does the system know to feed them with appropriate rates?
Or, more complex: my internet connection is PPPoE. The ppp daemon encapsulates the data and puts it onto an ethernet, that runs 100Mbit/sec. Then comes a DSL modem, and the DSL bandwidth is only about 2Mbit/sec. If the system would put out the data onto ethernet in full possible speed, the DSL modem would need to throw away 98% of the packets. It obviousely doesn't. But why?

 

[leebrown66]

I don't use openvpn, but googleing "openvpn rate limit" I found this:

Reference Manual For OpenVPN 2.4 | OpenVPN

Because OpenVPN tries to be a universal VPN tool offering a great deal of flexibility, there are a lot of options on this reference page for OpenVPN 2.4

openvpn.net

–shaper n Limit bandwidth of outgoing tunnel data to n bytes per second on the TCP/UDP port. Note that this will only work if mode is set to p2p. If you want to limit the bandwidth in both directions, use this option on both peers.OpenVPN uses the following algorithm to implement traffic shaping: Given a shaper rate of n bytes per second, after a datagram write of b bytes is queued on the TCP/UDP port, wait a minimum of (b / n) seconds before queuing the next write.​

​

​

​

It should be noted that OpenVPN supports multiple tunnels between the same two peers, allowing you to construct full-speed and reduced bandwidth tunnels at the same time, routing low-priority data such as off-site backups over the reduced bandwidth tunnel, and other data over the full-speed tunnel.​

​

​

Also note that for low bandwidth tunnels (under 1000 bytes per second), you should probably use lower MTU values as well (see above), otherwise the packet latency will grow so large as to trigger timeouts in the TLS layer and TCP connections running over the tunnel.​

​

​

OpenVPN allows n to be between 100 bytes/sec and 100 Mbytes/sec.​

 

[Eric A. Borisch]

Nope, they got dropped freely when unable to push anywhere along the network. Best-effort delivery, no guarantee. That’s fine for some applications. Consider a real-time temperature stream. If you don’t get time N, but you get N+1, that’s OK, and you don’t want to take the time to handle the retransmission if there is no benefit to the application.

Applications which require delivery reliability either build it above UDP with their own feedback, or use TCP (most common.)

 

[gpw928]

DSL bandwidth is only about 2Mbit/sec. If the system would put out the data onto ethernet in full possible speed, the DSL modem would need to throw away 98% of the packets. It obviousely doesn't. But why?

My gut reaction from working with firewalls was ICMP Source Quench (type 4). [Most firewalls take care to let these through.]

However, RFC 1812 says that "A router MAY ignore any ICMP Source Quench messages it receives". So no joy there. Well, maybe some, but not enough.

I think that the answer is that TCP has integrated flow control, and UDP does not. So applications using UDP must implement flow control, as required.

I suspect that many UDP applications aren't sufficiently complex as to need flow control (which is why they use UDP).

But UDP applications which do need flow control, and don't implement it, may have their packets silently dumped by intervening router(s).

 

[leebrown66]

Source Quench - Deprecated in RFC6633 (excerpt)

A host MUST NOT send ICMP Source Quench messages.​

​

If a Source Quench message is received, the IP layer MAY silently​

discard it.​

 

[PMc]

I don't use openvpn, but googleing "openvpn rate limit" I found this:
-shaper n Limit bandwidth of outgoing tunnel data to n bytes per second on the TCP/UDP port. Note that this

Yeah, that's what I'm currently doing.

It solves the problem, but it doesn't charm my perfection attitude: while one coulf tune the dummynet bandwidths dynamically due to demand, one cannot change that option alongside. (There might also be multiple tunnelled streams, and then one would have to distribute the bandwidth manually.)

BTW, there are more nice options in the openvpn - there are also -sndbuf and -rcvbuf knobs. While these did not help the issue, they seem to improve utilization:
openvpn_in[95433]: Socket Buffers: R=[42080->393216] S=[9216->393216]
No idea yet about the optimal values; my usual impression is that FreeBSD system defaults are either good or tuneable via sysctl, but this SO_SNDBUF appears to be neither.

 

[PMc]

Nope, they got dropped freely when unable to push anywhere along the network. Best-effort delivery, no guarantee. That’s fine for some applications. Consider a real-time temperature stream. If you don’t get time N, but you get N+1, that’s OK, and you don’t want to take the time to handle the retransmission if there is no benefit to the application.

Applications which require delivery reliability either build it above UDP with their own feedback, or use TCP (most common.)

Yes, that is exactly what I was taught.

But now we have these tunnelling applications like openvpn, and we do not want to encapsulate TCP into TCP. And since we cannot influence the inner protocols, we must resort to UDP for the outer protocol.
Also, in contrast to usual UDP services, these tunnelling and VPN services tend to produce serious traffic and saturate links. (There seems to be the need for a generally better adapted protocol for these.)

So probably the most "correct" approach would be to add the noerror option to dummynet (which should result in silent discard and rid us of the "no buffer space" messages in the logs), and then tune the buffers until the inner TCP prodocols manage to handle the flow control as good as possible.

 

[PMc]

My gut reaction from working with firewalls was ICMP Source Quench (type 4). [Most firewalls take care to let these through.]

However, RFC 1812 says that "A router MAY ignore any ICMP Source Quench messages it receives". So no joy there. Well, maybe some, but not enough.

These would be visible:

$ netstat -s
icmp:
        Output histogram:
                destination unreachable: 262065
        Input histogram:
                destination unreachable: 16759
                #7: 46227
                time exceeded: 26

I think that the answer is that TCP has integrated flow control, and UDP does not. So applications using UDP must implement flow control, as required.

Let's put it different: when we write to a file, that write will block until the disk gets the job done (at least in the old-fashioned way or when synchronous).
Now when we write to a socket... it appears to behave similar. There seems to be a kind of feedback at least between the network card's buffer and the socket, so when the buffer is full, the application will wait (given that the socket and the outgoing netif is in the same running kernel).
But when the packets go to a dummynet pipe, this does not happen, and error 55 is returned instead.

But UDP applications which do need flow control, and don't implement it, may have their packets silently dumped by intervening router(s).

Yes. because the "intervening router" has no means to talk back thru the network to the originating application. But within the same kernel there could be such a means.
So maybe there is one, and dummynet, because it was designet to emulate a network, does not implement it.

There is another interesting notion in the ipfw manpage: a dummynet pipe can adjust its bandwidth to the "transmit clock" of a netif. But when I try to do that, i get ipfw: no if support.
Will keep investigating...

 

[leebrown66]

I don't think Dummynet has any clue your application exists, has no sense of whether the traffic is local or remote. Consider your box is purely routing and another box has the application on it, dummynet on the routing box has no way to communicate with that other box's kernel or application.

Dummynet is acting like a network, as you say, but I think you're wanting it to act like a switch, which is directly connected.

In my mind a network is something I throw a packet into, traverses an unknown number of devices that may or may not drop/alter the packet, hope it makes it out the other end with no guarantee of order. I don't expect the entire chain to stall, waiting for my packets.

What you describe between the socket and kernel is more similar to a switch port. The NIC and switch port are directly connected, the receiver has a buffer that can fill, the NIC-port connection can block with pause frames at L2, although I rarely see this anymore because they tend to stuff overly large buffers into switches.

 

[PMc]

Dummynet is acting like a network, as you say, but I think you're wanting it to act like a switch, which is directly connected.

Well, what I actually want is a means to keep reachable via phone when something fills my network, while still keeping maximum thruput for the other data. I didn't want to have VoIP, but the TelCo left me no choice.
So now I'm figuring out how to do it best, but, most important, to fully understand as much as possible of the stuff, because if I already have to bother with the VoIP, I can as well try if I can route it onwards thru VPN.

The dummynet is just a tool built into my router machine, which can be useful for the task. I perceive it as a traffic shaper, not as a network emulation. Now, if the packets go thru dummynet unlimited and then are throttled on the WAN link, it seems to work well. But if the dummynet is throttling things, packets get dropped. And so I started wondering.
Maybe it is just a matter of a well crafted buffer configuration - then again this is something that can be found out.

What you describe between the socket and kernel is more similar to a switch port. The NIC and switch port are directly connected, the receiver has a buffer that can fill, the NIC-port connection can block with pause frames at L2, although I rarely see this anymore because they tend to stuff overly large buffers into switches.

I don't use switches or routers, only FreeBSD machines. And I can see such pause frames on my lan connections (FreeBSD to FreeBSD):

$ ifconfig fxp4
       media: Ethernet autoselect <flowcontrol>
$ sysctl dev.fxp.4.stats.tx.pause
dev.fxp.4.stats.tx.pause: 26006

 

[PMc]

It appears the effect is mostly solveable by carefull buffer adjustment:
use a small buffer on the pipe and a rather large one on the queues for the UDP.
For 1.6 Mbit link speed I am now using:

pipe 1 config bw 1500kbit/s  queue 7  sched config type qfq
queue 11 config pipe 1 weight 1 mask all buckets 2048 maxlen 1492 queue 20Kbytes droptail # general
queue 12 config pipe 1 weight 95 mask all maxlen 1492 queue 20Kbytes droptail # hi-prio
queue 13 config pipe 1 weight 20 maxlen 1492 queue 200Kbytes droptail noerror # UDP-bulk

Packet loss is now 0.05%, and the bandwith limiting works:
fully saturated UDP traffic without the pipe:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+syrte8.obspm.fr 145.238.203.14   2 u   56   64   37   82.925  -14.813  65.028
+schnitzel.team  193.190.230.66   2 u   47   64   37  144.737   20.626  42.963
+mail.stygium.ne 131.188.3.223    2 u   52   64   37   22.360  -35.984  53.735
+stratum2-4.NTP. 129.70.130.70    2 u   56   64   37   80.865  -14.314  80.991
+time3.hs-augsbu 131.188.3.222    2 u   31   64   37   80.514   -5.579  93.759
*atlas.linocomm. 130.149.17.21    2 u   32   64   37   75.619   -4.747  65.396
+217.144.138.234 124.216.164.14   2 u   29   64   37  102.599   11.187  66.978

with the pipe:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+syrte8.obspm.fr 145.238.203.14   2 u   29   64   77   33.492   -4.562   4.853
-schnitzel.team  193.79.237.14    2 u   21   64   77   32.024   -5.007   5.120
-mail.stygium.ne 131.188.3.223    2 u   26   64   77   20.053   -3.494   6.447
+stratum2-4.NTP. 129.70.130.71    2 u   23   64   77   29.791   -4.637   4.982
*time3.hs-augsbu 131.188.3.222    2 u    8   64   77   33.041   -3.496   3.097
-atlas.linocomm. 130.149.17.21    2 u    1   64   77   25.594  -10.435   6.480
-217.144.138.234 124.216.164.14   2 u    7   64   77   25.611   -3.764   4.480