IPFW + Dummynet: Adding a delay decreases iperf3 transfer speed significantly

T

tkzheng

Hi All,

I've sent so much time on this that my head is pounding. Here is some context:

I am running FreeBSD on a box with two ethernet NICs, re0 and re1. I have a second machine connected to the first ethernet interface and a third machine connected to the second ethernet interface. My goal is to use FreeBSD to act as a router between the two, using dummynet to introduce additional bandwidth, latency and packet loss restrictions.

Now as far as I can tell, I have successfully set everything up, including ipfw and dummynet. My two machines grab IP addresses from the FreeBSD machine using DHCP supplied by DNSMASQ. I have also configured a bridge between the two interface on the FreeBSD machine. Iperf3 runs successfully both ways, up to 940 Mbit/s since I'm using gigabit connections.

Now the issue. First I create a pipe using the following:
Code: 
ipfw add 3000 pipe 1 ip from any to any in
ipfw config pipe 1 bw 100Mbit/s

Running iperf3 between the two machines, I get the expected bandwidth of around 95Mbit/s as show below:
Code: 
iperf3 -c 192.168.0.106 -t 30
Connecting to host 192.168.0.106, port 5201
[  4] local 192.168.0.93 port 41774 connected to 192.168.0.106 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  12.0 MBytes   101 Mbits/sec   37   83.4 KBytes       
[  4]   1.00-2.00   sec  11.4 MBytes  95.4 Mbits/sec   15   74.9 KBytes       
[  4]   2.00-3.00   sec  11.4 MBytes  95.4 Mbits/sec    7   74.9 KBytes       
[  4]   3.00-4.00   sec  11.4 MBytes  95.4 Mbits/sec   10   74.9 KBytes       
[  4]   4.00-5.00   sec  11.4 MBytes  95.4 Mbits/sec    6   82.0 KBytes       
[  4]   5.00-6.00   sec  11.4 MBytes  95.9 Mbits/sec    9   89.1 KBytes       
[  4]   6.00-7.00   sec  11.4 MBytes  95.4 Mbits/sec    7   70.7 KBytes       
[  4]   7.00-8.00   sec  11.3 MBytes  94.9 Mbits/sec   13   67.9 KBytes       
[  4]   8.00-9.00   sec  11.4 MBytes  95.4 Mbits/sec    5   79.2 KBytes       
[  4]   9.00-10.00  sec  11.3 MBytes  94.9 Mbits/sec   13   72.1 KBytes       
[  4]  10.00-11.00  sec  11.4 MBytes  95.4 Mbits/sec    8   83.4 KBytes       
[  4]  11.00-12.00  sec  11.4 MBytes  95.9 Mbits/sec   17   74.9 KBytes       
[  4]  12.00-13.00  sec  11.4 MBytes  95.4 Mbits/sec   12   67.9 KBytes       
[  4]  13.00-14.00  sec  11.4 MBytes  95.9 Mbits/sec    8   74.9 KBytes       
[  4]  14.00-15.00  sec  11.3 MBytes  94.9 Mbits/sec   13   73.5 KBytes       
[  4]  15.00-16.00  sec  11.4 MBytes  95.4 Mbits/sec   18   65.0 KBytes       
[  4]  16.00-17.00  sec  11.4 MBytes  95.9 Mbits/sec    6   76.4 KBytes       
[  4]  17.00-18.00  sec  11.4 MBytes  95.4 Mbits/sec    6   80.6 KBytes       
[  4]  18.00-19.00  sec  11.4 MBytes  95.4 Mbits/sec   12   70.7 KBytes       
[  4]  19.00-20.00  sec  11.4 MBytes  95.4 Mbits/sec   12   74.9 KBytes       
[  4]  20.00-21.00  sec  11.4 MBytes  95.4 Mbits/sec   15   76.4 KBytes       
[  4]  21.00-22.00  sec  11.4 MBytes  95.4 Mbits/sec   12   74.9 KBytes       
[  4]  22.00-23.00  sec  11.4 MBytes  95.9 Mbits/sec   17   72.1 KBytes       
[  4]  23.00-24.00  sec  11.4 MBytes  95.4 Mbits/sec   12   69.3 KBytes       
[  4]  24.00-25.00  sec  11.4 MBytes  95.4 Mbits/sec   10   73.5 KBytes       
[  4]  25.00-26.00  sec  11.4 MBytes  95.4 Mbits/sec   12   66.5 KBytes       
[  4]  26.00-27.00  sec  11.4 MBytes  95.4 Mbits/sec    6   70.7 KBytes       
[  4]  27.00-28.00  sec  11.4 MBytes  95.4 Mbits/sec   14   66.5 KBytes       
[  4]  28.00-29.00  sec  11.4 MBytes  95.4 Mbits/sec    8   70.7 KBytes       
[  4]  29.00-30.00  sec  11.4 MBytes  95.4 Mbits/sec   10   69.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec   342 MBytes  95.6 Mbits/sec  350             sender
[  4]   0.00-30.00  sec   341 MBytes  95.4 Mbits/sec                  receiver

iperf Done.

I then configure the pipe with a 15ms delay, creating a 30ms RTT between the two machines running iperf.
Code: 
ipfw config pipe 1 bw 100Mbit/s delay 15ms

For some reason, the bandwidth decreases to around 70Mbit/s. This huge decrease in bandwidth is not explainable by anything I understand, including the TCP slow start period and the different congestion control periods. I've even tried an interval of up to 90s to check the maximum transfer speed. The result is below.
Code: 
iperf3 -c 192.168.0.106 -t 30
Connecting to host 192.168.0.106, port 5201
[  4] local 192.168.0.93 port 41770 connected to 192.168.0.106 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  6.82 MBytes  57.2 Mbits/sec  101    189 KBytes       
[  4]   1.00-2.00   sec  6.03 MBytes  50.6 Mbits/sec    0    211 KBytes       
[  4]   2.00-3.00   sec  6.71 MBytes  56.3 Mbits/sec    0    233 KBytes       
[  4]   3.00-4.00   sec  7.39 MBytes  62.0 Mbits/sec    0    256 KBytes       
[  4]   4.00-5.00   sec  8.02 MBytes  67.2 Mbits/sec    0    280 KBytes       
[  4]   5.00-6.00   sec  8.51 MBytes  71.4 Mbits/sec    0    303 KBytes       
[  4]   6.00-7.00   sec  9.26 MBytes  77.7 Mbits/sec    0    325 KBytes       
[  4]   7.00-8.00   sec  9.69 MBytes  81.3 Mbits/sec    0    348 KBytes       
[  4]   8.00-9.00   sec  8.14 MBytes  68.3 Mbits/sec    1    286 KBytes       
[  4]   9.00-10.00  sec  7.77 MBytes  65.2 Mbits/sec    4    226 KBytes       
[  4]  10.00-11.00  sec  7.15 MBytes  59.9 Mbits/sec    0    247 KBytes       
[  4]  11.00-12.00  sec  7.64 MBytes  64.1 Mbits/sec    0    267 KBytes       
[  4]  12.00-13.00  sec  8.14 MBytes  68.3 Mbits/sec    0    290 KBytes       
[  4]  13.00-14.00  sec  6.96 MBytes  58.4 Mbits/sec    1    239 KBytes       
[  4]  14.00-15.00  sec  7.71 MBytes  64.6 Mbits/sec    0    267 KBytes       
[  4]  15.00-16.00  sec  8.26 MBytes  69.3 Mbits/sec    0    284 KBytes       
[  4]  16.00-17.00  sec  8.64 MBytes  72.5 Mbits/sec    0    291 KBytes       
[  4]  17.00-18.00  sec  8.95 MBytes  75.1 Mbits/sec    0    314 KBytes       
[  4]  18.00-19.00  sec  8.26 MBytes  69.3 Mbits/sec    2    250 KBytes       
[  4]  19.00-20.00  sec  8.08 MBytes  67.8 Mbits/sec    0    287 KBytes       
[  4]  20.00-21.00  sec  8.76 MBytes  73.5 Mbits/sec    0    308 KBytes       
[  4]  21.00-22.00  sec  9.26 MBytes  77.7 Mbits/sec    0    320 KBytes       
[  4]  22.00-23.00  sec  9.45 MBytes  79.2 Mbits/sec    0    327 KBytes       
[  4]  23.00-24.00  sec  9.82 MBytes  82.4 Mbits/sec    0    351 KBytes       
[  4]  24.00-25.00  sec  9.45 MBytes  79.2 Mbits/sec    1    270 KBytes       
[  4]  25.00-26.00  sec  8.64 MBytes  72.5 Mbits/sec    0    314 KBytes       
[  4]  26.00-27.00  sec  8.51 MBytes  71.4 Mbits/sec    9    242 KBytes       
[  4]  27.00-28.00  sec  7.64 MBytes  64.1 Mbits/sec    0    266 KBytes       
[  4]  28.00-29.00  sec  8.14 MBytes  68.3 Mbits/sec    0    284 KBytes       
[  4]  29.00-30.00  sec  6.90 MBytes  57.9 Mbits/sec    1    238 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec   245 MBytes  68.4 Mbits/sec  120             sender
[  4]   0.00-30.00  sec   243 MBytes  67.9 Mbits/sec                  receiver

iperf Done.

This has been extremely perplexing for me. The only thing different between the two iperf3 outputs is the consistency of the Retr column, which I believe indicates how many retransmission there were. The consistency of Retr without the 15ms delay is higher, averaging around 10 retransmissions per second. The Retr of the pipe with a 15ms delay confusingly averages 0 retransmissions, but every time there is any, even if it is 3 per second, the speed immediately decreases.

Any help in understanding what is happening is greatly appreciated.
 
Could you show us your interface configs?
ifconfig of the bridge interface and re0, re1.

Also, if you want the FreeBSD box to route packets between the other two machines, why are you bridging the interfaces? It should simply do IP forwarding.
 
Another idea, some people have written in other topics about performance problems with IPFW and if I recall correctly, the solution was to deactivate some of the unused flags on the network interfaces. I think this was due to a kernel bug.
Sorry, I can't find the post anymore.
 
I forgot to mention, but there is actually a third network interface, wlan0, that I use to connect to my home wifi from the FreeBSD machine, and the FreeBSD machine also does NAT for the two physically connected machines. I don't know if that will make any difference. My config is show below.

Code: 
ifconfig_wlan0="WPA SYNCDHCP"
defaultrouter="10.0.1.1"

gateway_enable="YES"
firewall_enable="YES"
firewall_type="OPEN"
natd_enable="YES"
natd_interface="wlan0"
natd_flags="-f /etc/natd.conf"
dnsmasq_enable="YES"

ifconfig_re0="inet 192.168.0.1 netmask 255.255.255.0"
ifconfig_re1="inet 192.168.0.2 netmask 255.255.255.0"

cloned_interfaces="bridge0"
ifconfig_bridge0="addm re0 addm re1 up"

You mentioned IP forwarding. Could you point me in the direction for this? I'm completely new to this and this is actually my first time using FreeBSD.

Thanks
 
tkzheng said:
You mentioned IP forwarding. Could you point me in the direction for this? I'm completely new to this and this is actually my first time using FreeBSD.

Thanks
Click to expand...

Well this is general networking. It is not FreeBSD specific.

Bridging forwards Ethernet frames on the Data Link layer (layer 2), while IP forwarding does something similar but for IP datagrams on the Network Layer (layer 3).
Bridging is similar to what a network switch does and routing is similar to what your Internet router does. A switch could combine two network segments into one subnet. The router stand between subnets and forwards IP datagrams between them.

You need to clarify which of the two you want to do, but in general you don't want to do both (at least I am not aware of such scenario). It would be like putting a network switch and a router in parallel to both bridge AND route between two networks. Weird.

So in your configuration the "ifconfig_bridge0" line bridges re0 and re1 and the lines "gateway_enable", "natd_enable" implement routing via the interface wlan0.

What seems a bit weird to me is you set two separate IP addresses on re0 and re1 but then bridge them together.
What exactly are you trying to achieve? If the FreeBSD box needs to be a router, you could remove the bridging.

Do you want specifically to enable Ethernet communication between the two other machines? If not, don't bridge, simply enable IP forwarding via sysctl net.inet.ip.fw.enable=1 (put it also in sysctl.conf) and sysrc gateway_enable="YES" and configure NAT in your firewall.
 
If on the other hand you want to create a LAN from your two other machines, then leave the bridge but don't set two IP addresses on re0 and re1. Leave them without IP addresses and set a single IP address on the bridge itself:

Code: 
ifconfig_re0="up"
ifconfig_re1="up"

cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0 addm re0 addm re1 up"

Also, your other two machines need IP addresses: 192.168.0.2 and 192.168.0.3 for example. They could get them via dnsmasq, as I see you have enabled it, or configure them statically, it does not matter.

IP forwarding is required also in this scenario, but only to the outside via interface wlan0.

In this case wlan0 should have an IP address in another subnet, for example 10.0.1.2 or any other that is not 192.168.0.x. wlan0 is not bridged with re0 and re1, so you need to route and it needs a separate subnet.
 
This setup does seem to make much more sense to me, thanks! I have switched over to giving the bridge an IP, and not configuring re0 and re1.

This however, doesn't seem to fix the bandwidth issue. I'm going to try to look for what you mentioned about the flags.
 
tkzheng said:
This setup does seem to make much more sense to me, thanks! I have switched over to giving the bridge an IP, and not configuring re0 and re1.

This however, doesn't seem to fix the bandwidth issue. I'm going to try to look for what you mentioned about the flags.
Click to expand...
The IP addressing should not have an effect on the bandwidth. When you put two IP addresses on re0 and re1 and bridge them in effect you give the FreeBSD box two IP addresses on the bridge, which is valid but I don't see usage for it.
Regarding the bandwidth, as I wrote above, there was a forum topic where somebody reported that turning off some flags made his NAT faster (with IPFW and FreeBSD 12) and they wrote it was a bug.
Unfortunately I cannot find the post anymore, so search for it maybe you get lucky. In the post they mentioned exactly which flags helped in their case.

Something else to try, if this does not help, try removing the bridge and using only IP forwarding + NAT. This would skip a lot of the Level 2 overhead and may have the same effect as removing the pesky flags.

Good luck.

Edit: I found an interesting paper about dummynet bandwidth with IPFW. Maybe it could shed some light for you: www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf
 
Thanks for all the help. Unfortunately again it doesn't seem to be my issue. Playing with the flags seem to have no use, both in the network interface as well as using sysctl. I'll keep looking.

On another note, do you know of any way I can run dummynet without IPFW or maybe a different network emulator that plays nice with FreeBSD?
 
IPFW ist just one of the firewalls. You could try with the other two. I don't know anything about them though.
 
tkzheng said:
Any help in understanding what is happening is greatly appreciated.
Click to expand...

Short suggestion: set up an additional asynchronous pipe between the router and each of your boxes and, reroute upstream traffic in one of the pipe and downstream traffic in the other pipe between the router and each of your boxes.

Subject to the traffic in your network, you may want to invest in four additional interfaces (one for each of the boxes and two for the router) and consider bridging as a supplementary option.

Boring explanation: traffic shapping is like fluid dynamics. There was an excellent article on PF and ALTQ that could help you understand my point (Prioritizing empty TCP ACKs with pf and ALTQ, it seems to have been taken off line but there is a cache on Google).

I know that ALTQ is using primarily a time-division approach while DummyNet a channel access one. Because of this, I think you need to set up a second lane for the traffic the other way around.
 
You must log in or register to reply here.
Back
Top