Reply (Howto: traffic shaping VOIP and data with pf and dummynet prio)

[PMc]

[Mod: Split off from: https://forums.freebsd.org/threads/...oip-and-data-with-pf-and-dummynet-prio.103775]

Nice. I have shaping activated, but not sure if it helps anything - I didn't notice problems in voice quality, which could mean that there are none, or that the traffic shaper is good enough, or that the network isn't saturated when I do voice calls.

$f pipe 1 config bw 6250kbit/s queue 28Kbytes mask $nomask buckets 2048 sched config type qfq # WAN uplink
$f pipe 2 config bw 20200kbit/s queue 25 mask $nomask buckets 2048 sched config type qfq # WAN dwnlnk

I am not sure what all these options do, how they influence each other or how that could be effectively tested.

Issue: you need to know your technical saturation limit, and stay some 3% below, so that there is actually a bulge of data before the pipe from which prioritizing can happen. This is fine with a hard-clocked link (constant bw), but DSL isn't hard-clocked; you get some bw that is dependent on current line-interferences and whatever.

Issue: clocking (kern.hz). dummynet doesn't behave well when that is put low. But setting kern.hz to some 2000 or 4000 can significantly change energy consumption. And this is not for a gaming rig latency, it is for 24/7 ops. Not nice.

# Outbound gen.
$f queue 11 config pipe 1 weight 1 mask $outmask buckets 2048 maxlen 1492 queue 32Kbytes droptail
# Outbound prio (VoIP)
$f queue 12 config pipe 1 weight 95 mask $outmask buckets 64 maxlen 1492 queue 20Kbytes droptail
# Outbound bulk
$f queue 13 config pipe 1 weight 20 mask $outmask buckets 64 maxlen 1492 queue 32Kbytes droptail noerror

noerror: some things gave back "UDP: no buffer space" when the dummynet would throttle them, but they would not give back errors when the uplink would throttle them. Never found time to figure out why (would need a deep-dive into network stuff).

When I try to reconfigure at runtime, I usually get kernel crash: qfq_dequeue BUG/* non-workconserving leaf.

At some point it all went over my head. Here are some bad inter-provider peerings, I am doing long-haul (>200ms) mass-data transfers, TCP congestion-control does also influence things (and has/had bugs), so I really don't know where to start to do comparable measurements.
And then the telco switched to VDSL, and 90% of the painful problems just went away. Even backup over VPN (mtu=1500, have the kernel fragment everything) just works. Even VPN inside VPN (have the kernel triple-fragment everything) gets somehow along.
So for now this matter is left as it is, until some day I might find the mood to read the source and try to really understand the ideas of the design (documentation is unsat).

• dummynet has queues 1 and 3 for voice and 2 and 4 for data traffic. A prio scheduler is used, and it uses queue weight to determine priority, with lower weight meaning higher priority. The voice queues have weight 0, data queues weight 1.

Strange. I thought higher weight means higher prio, and only values 1-100 are allowed.

 

[dj015]

[Mod: Split off from: https://forums.freebsd.org/threads/...oip-and-data-with-pf-and-dummynet-prio.103775]

Nice. I have shaping activated, but not sure if it helps anything - I didn't notice problems in voice quality, which could mean that there are none, or that the traffic shaper is good enough, or that the network isn't saturated when I do voice calls.

$f pipe 1 config bw 6250kbit/s queue 28Kbytes mask $nomask buckets 2048 sched config type qfq # WAN uplink
$f pipe 2 config bw 20200kbit/s queue 25 mask $nomask buckets 2048 sched config type qfq # WAN dwnlnk

I am not sure what all these options do, how they influence each other or how that could be effectively tested.

Issue: you need to know your technical saturation limit, and stay some 3% below, so that there is actually a bulge of data before the pipe from which prioritizing can happen. This is fine with a hard-clocked link (constant bw), but DSL isn't hard-clocked; you get some bw that is dependent on current line-interferences and whatever.

Issue: clocking (kern.hz). dummynet doesn't behave well when that is put low. But setting kern.hz to some 2000 or 4000 can significantly change energy consumption. And this is not for a gaming rig latency, it is for 24/7 ops. Not nice.

# Outbound gen.
$f queue 11 config pipe 1 weight 1 mask $outmask buckets 2048 maxlen 1492 queue 32Kbytes droptail
# Outbound prio (VoIP)
$f queue 12 config pipe 1 weight 95 mask $outmask buckets 64 maxlen 1492 queue 20Kbytes droptail
# Outbound bulk
$f queue 13 config pipe 1 weight 20 mask $outmask buckets 64 maxlen 1492 queue 32Kbytes droptail noerror

noerror: some things gave back "UDP: no buffer space" when the dummynet would throttle them, but they would not give back errors when the uplink would throttle them. Never found time to figure out why (would need a deep-dive into network stuff).

That's a very FreeBSD-specific issue, on eg. Linux, when the outgoing UDP socket buffer is full, ENOBUFS is never returned to user space, the packet is silently dropped instead. If it's caused by bursts, you might be able to increase the queue size to prevent it, but if continuously sending at a faster rate than the link can handle, ENOBUFS is inevitable.

When I try to reconfigure at runtime, I usually get kernel crash: qfq_dequeue BUG/* non-workconserving leaf.

At some point it all went over my head. Here are some bad inter-provider peerings, I am doing long-haul (>200ms) mass-data transfers, TCP congestion-control does also influence things (and has/had bugs), so I really don't know where to start to do comparable measurements.
And then the telco switched to VDSL, and 90% of the painful problems just went away. Even backup over VPN (mtu=1500, have the kernel fragment everything) just works. Even VPN inside VPN (have the kernel triple-fragment everything) gets somehow along.
So for now this matter is left as it is, until some day I might find the mood to read the source and try to really understand the ideas of the design (documentation is unsat).

There is a lot of documentation, but it's scattered and somewhat outdated.

The dnctl(8) man page is terrible and desperately needs updates. It has serious mistakes: "In addition to the type, all parameters allowed for a pipe can also be specified for a scheduler" is completely wrong, you cannot specify "bw", "delay", "burst" for a scheduler, I wonder how a statement that wrong ever got written, maybe it meant to say "queue" instead of "pipe"? It doesn't document the PRIO scheduler, and my patch for doing so is getting ignored (https://github.com/freebsd/freebsd-src/pull/2412) . Also it uses ambiguous terms like "link" without explaining them.

/usr/src/sys/netpfil/ipfw/dummynet.txt is great at describing the internals.
/usr/include/netinet/ip_dummynet.h is also well commented and good for understanding data structures.
/usr/src/sys/netpfil/ipfw/ip_dn_private.h is great for in-kernel data structures, which also use the ones in /usr/include/netinet/ip_dummynet.h.

https://web.archive.org/web/20240224032114/http://info.iet.unipi.it/~luigi/dummynet was the original dummynet website, with links to many resources.
https://web.archive.org/web/2024052...t.unipi.it/~luigi/doc/20100513-bsdcan10dn.pdf is an excellent presentation about Dummynet, with great diagrams and descriptions of internals.

Strange. I thought higher weight means higher prio, and only values 1-100 are allowed.

The scheduler is responsible for dealing with weight, and can interpret it any way it likes. WFQ2 might treat larger weights as higher priorities, but it's the opposite for PRIO.

 

[loveydovey]

The scheduler is responsible for dealing with weight, and can interpret it any way it likes. WFQ2 might treat larger weights as higher priorities, but it's the opposite for PRIO.

And it seems some algos like fq_codel ignore the weights entirely by design - it just equally spreads the bandwidth across flows using an equal share approach (round-robin). (Is this true?) You can control how much of a share per pass fq_codel gives with quantum (giving larger byte allocation to a flow per pass and thus increasing relative weight during contention).

 

[dj015]

And it seems some algos like fq_codel ignore the weights entirely by design - it just equally spreads the bandwidth across flows using an equal share approach (round-robin). (Is this true?) You can control how much of a share per pass fq_codel gives with quantum (giving larger byte allocation to a flow per pass and thus increasing relative weight during contention).

Yes, in fact /usr/src/sys/netpfil/ipfw/dn_sched_fq_codel.c uses struct dn_extra_parms with its:

#define DN_MAX_EXTRA_PARM 10
int64_t par[DN_MAX_EXTRA_PARM];

instead of the struct dn_fs 32 bit par field of which there is only 4, with the first being weight:

/* generic scheduler parameters. Leave them at -1 if unset.
* Now we use 0: weight, 1: lmax, 2: priority
*/
int par[4];