OpenVPN bridge not working on L2 or very slow on L3

marwis · Nov 12, 2013

I am experiencing strange problems with bridging on 9.1-RELEASE with the GENERIC kernel running on ESXi 5.0.

I have a fairly standard bridged security/openvpn server listening on vmx3f0 on port UDP/1194. The traffic leaves on the tap0 interface which is bridged to the vmx3f1 interface.

Code:

+-------FreeBSD 9.1 on ESXi 5.0----------+
|                                        |
|                         +-------+      |    +-+
|                      +--+bridge0+--+   |    |v|
|                      |  +-------+  |   |    |S|
|                      |             |   |    |w|
+------+             +-+--+       +--+---+    |i|
|vmx3f0+==#openvpn#--+tap0|       |vmx3f1+----+t|
+------+             +----+       +------+    |c|
|                                        |    |h|
+----------------------------------------+    +-+

Both bridged interfaces tap0 and vmx3f1 are added to the bridge in a standard way.

# ifconfig bridge0 create
# ifconfig tap0 create
# ifconfig bridge0 addm tap0 addm vmx3f1 up
# ifconfig vmx3f1 up
# ifconfig bridge0

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:92:09:21:cc:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vmx3f1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 31250000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000

Since the tap0 interface is down, we make it go up when OpenVPN opens it. # sysctl net.link.tap.up_on_open=1 # default 0

Filtering is disabled on bridges.
# sysctl net.link.bridge.pfil_member=0 # default 1
# sysctl net.link.bridge.pfil_bridge=0 # default 1

OpenVPN is configured and started this way with # openvpn --config openvpn.conf where
# cat openvpn.conf

Code:

server-bridge 192.0.2.49 255.255.255.240 192.0.2.59 192.0.2.62
local 192.0.2.126
lport 1194
proto udp
persist-tun
dev tap0
keepalive 5 10
cd /usr/local/etc/openvpn
script-security 3
suppress-timestamps
verb 4
push "dhcp-option DNS 198.51.100.10"
push "dhcp-option WINS 198.51.100.10"
push "dhcp-option DNS 198.51.100.18"
push "dhcp-option WINS 198.51.100.18"
# routes
push "route 198.51.100.74 255.255.255.255 192.0.2.49"
push "route 198.51.100.76 255.255.255.255 192.0.2.49"
auth-user-pass-verify "scripts/krb-authenticate.sh" via-env
client-cert-not-required
username-as-common-name
cipher none
tls-server
tls-cipher TLS-RSA-WITH-AES-128-CBC-SHA
ca /usr/local/etc/openvpn/ssl/certs/ca_authorities.crt
dh /usr/local/etc/openvpn/ssl/keys/openvpn-dh.pem
cert /usr/local/etc/openvpn/ssl/certs/openvpn-tls.crt
key /usr/local/etc/openvpn/ssl/keys/openvpn-tls.key
persist-key

Note: Encryption is disabled with cipher none for testing so it doesn't influence the performance.

When an OpenVPN client connects, it is assigned IP address 192.0.2.59 but its MAC address is not added to the list of addresses known to bridge0:
# ifconfig bridge0 addr | grep tap0

Code:

# (empty)

If I arping() the client from a different host connected to the vSwitch arping 192.0.2.59, the OpenVPN client's MAC address appears in the bridge0's list:
# ifconfig bridge0 addr | grep tap0

Code:

00:bd:c8:e1:b9:00 Vlan1 tap0 1199 flags=0<>

The problem is however, the address is forgotten as soon as I start ping()ing the gateway from the OpenVPN client ping 192.0.2.49.

# ifconfig bridge0 addr | grep tap0

Code:

# (empty)

I tried a (not too nice) workaround: do not let bridge0 forget the MAC addresses learned on tap0 by making it sticky.

# ifconfig bridge0 sticky tap0

The OpenVPN client is now fully functional, but there are major performance problems. If the client communicates with hosts on the same L2 network/vSwitch (i.e. 192.0.2.48/28), the bandwidth of the tunnel approaches 400 Mbit/s. However, when the client tries to communicate with a host on L3 (IP layer) meaning it has to use the default gateway 192.2.0.49 behind the tunnel, the traffic drops to approximately 80-90 Mbit/s for FreeBSD clients and to approximately 40 Mbit/s for Windows clients.

I did try to play with kernel options
# sysctl kern.ipc.maxsockbuf=16777216 # default 2097152
# sysctl net.inet.tcp.sendbuf_max=16777216 # default 2097152
# sysctl net.inet.tcp.recvbuf_max=16777216 # default 2097152
# sysctl net.inet.udp.recvspace=262144 # default 42080
and OpenVPN options

Code:

sndbuf 8388608   # default 65536
rcvbuf 8388608   # default 65536
fragment 1400
mssfix

but this did not help.

I'd appreciate any hint on what I might be doing wrong. Thanks!

marwis · Nov 13, 2013

marwis said:
If the client communicates with hosts on the same L2 network/vSwitch (i.e. 192.0.2.48/28), the bandwidth of the tunnel approaches 400 Mbit/s. However, when the client tries to communicate with a host on L3 (IP layer) meaning it has to use the default gateway 192.2.0.49 behind the tunnel, the traffic drops to approximately 80-90 Mbit/s for FreeBSD clients and to approximately 40 Mbit/s for Windows clients.

I didn't use the same technique to measure the bandwidth for L2 and for L3. When using iperf(), I'm getting approximately 400 Mbit/s for both. When downloading a file from a webserver using fetch(), the bandwidth drops to 100 Mbit/s.

I am still under the impression, however, that bridged OpenVPN should be able to handle more than 400 MBit/s. Linux, for instance, handles 1.3-3.8 Gbit/s for a physical tunneled connection. Can the virtualization penalty be this high?

marwis · Nov 15, 2013

marwis said:
When an OpenVPN client connects, it is assigned IP address 192.0.2.59 but its MAC address is not added to the list of addresses known to bridge0:
# ifconfig bridge0 addr | grep tap0

Code:

# (empty)

If I arping() the client from a different host connected to the vSwitch arping 192.0.2.59, the OpenVPN client's MAC address appears in the bridge0's list:
# ifconfig bridge0 addr | grep tap0

Code:

00:bd:c8:e1:b9:00 Vlan1 tap0 1199 flags=0<>

The problem is however, the address is forgotten as soon as I start ping()ing the gateway from the OpenVPN client ping 192.0.2.49.

# ifconfig bridge0 addr | grep tap0

Code:

# (empty)

For the record: it turned out the highly available default gateway behind the switch and/or the vSwitch itself is probably misconfigured. When the OpenVPN client was looking for the gateway by sending an ARP request (L2 broadcast), the same packet came back on the same interface. This confused the bridge and it started to think the OpenVPN client's MAC address is reachable via the interface vmx3f1, not tap0.

As a workaround, I marked the vmx3f1 interface as "non-learning" by # ifconfig bridge0 -learn vmx3f1. While the bridge now doesn't known (learn) the MAC address of the gateway is reachable via the vmx3f1 interface, the L2 unicast and broadcast frames are now forwarded with no problems.