Howto: traffic shaping VOIP and data with pf and dummynet prio

[dj015]

Mixing VOIP and data usually requires traffic shaping, to preserve voice quality at the cost of lower data throughput during phone calls.

This is how I did it on FreeBSD:

• Asterisk dialplan plays an IVR for testing call quality, configured to set TOS bits in the IP header for RTP to "ef" (Expedited Forwarding).
• pf as the firewall, sends UDP packets marked with "ef" to dummynet queues 1 for inbound and 3 for outbound, those not marked (ordinary data) to queue 2 for inbound and queue 4 for outbound.
• dummynet has queues 1 and 3 for voice and 2 and 4 for data traffic. A prio scheduler is used, and it uses queue weight to determine priority, with lower weight meaning higher priority. The voice queues have weight 0, data queues weight 1. The scheduler moves data into a pipe, throttled to only 10 Mbits/s for testing.
• Testing then involves doing these in parallel:
  • "ssh user@server cat /dev/zero > /dev/null" in 3-4 tabs at the same time, to saturate the network with TCP traffic.
  • phone call to Asterisk to listen to the audio through this very busy network.

Without traffic shaping, the packet loss, jitter and delay lead to terrible sound quality, tcpdump shows RTP packet loss as high as 33%.

But when properly configured:

/etc/rc.conf

pf_enable="YES"
dnctl_enable="YES"

/etc/dnctl.conf

## WAN
# downlink
pipe  1 config bw 10Mbits/s
sched 1 config pipe 1 type prio
queue 1 config sched 1 weight 0
queue 2 config sched 1 weight 1
# uplink
pipe  2 config bw 10Mbits/s
sched 2 config pipe 2 type prio
queue 3 config sched 2 weight 0
queue 4 config sched 2 weight 1

/etc/pf.conf

pass out quick on em0 tos ef dnqueue (3, 1)         
pass out on em0 dnqueue (4, 2) 
pass in quick on em0 tos ef dnqueue (1, 3)         
pass in on em0 dnqueue (2, 4)

Asterisk's pjsip endpoint needed this to mark the RTP packets' TOS so pf could send them to the right queue:

tos_audio=ef

With these settings, there was zero lost voice packets, low latency, and perfect sound quality. Also "dnctl queue show" shows how the dropped packets ("Drp") are only on the data queue (4), never on the voice queues (1 and 3):

q00001  50 sl. 1 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       27     5400  0    0   0
q00002  50 sl. 1 flows (1 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0      341    13784  0    0   0
q00003  50 sl. 1 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0       20     4000  0    0   0
q00004  50 sl. 1 flows (1 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
  0 ip           0.0.0.0/0             0.0.0.0/0     39721 63180862 50 77920 302

I also discovered that this prio scheduler is completely undocumented, since 2010, and have a PR pending on Github to add some documentation for it: https://github.com/freebsd/freebsd-src/pull/2412

It took me hours, and required reading its source code and header files, to understand that its

int prio = q->fs->fs.par[0];

was referring to "weight" in /usr/include/netinet/ip_dummynet.h:

        /* generic scheduler parameters. Leave them at -1 if unset.
         * Now we use 0: weight, 1: lmax, 2: priority
         */
        int par[4];

instead of the misleading "pri" command line parameter which wasn't working.