I am experiencing strange problems with bridging on 9.1-RELEASE with the GENERIC kernel running on ESXi 5.0.
I have a fairly standard bridged security/openvpn server listening on vmx3f0 on port UDP/1194. The traffic leaves on the tap0 interface which is bridged to the vmx3f1 interface.
Both bridged interfaces tap0 and vmx3f1 are added to the bridge in a standard way.
Since the tap0 interface is down, we make it go up when OpenVPN opens it.
Filtering is disabled on bridges.
OpenVPN is configured and started this way with
Note: Encryption is disabled with cipher none for testing so it doesn't influence the performance.
When an OpenVPN client connects, it is assigned IP address 192.0.2.59 but its MAC address is not added to the list of addresses known to bridge0:
If I arping() the client from a different host connected to the vSwitch
The problem is however, the address is forgotten as soon as I start ping()ing the gateway from the OpenVPN client
I tried a (not too nice) workaround: do not let bridge0 forget the MAC addresses learned on tap0 by making it sticky.
The OpenVPN client is now fully functional, but there are major performance problems. If the client communicates with hosts on the same L2 network/vSwitch (i.e. 192.0.2.48/28), the bandwidth of the tunnel approaches 400 Mbit/s. However, when the client tries to communicate with a host on L3 (IP layer) meaning it has to use the default gateway 192.2.0.49 behind the tunnel, the traffic drops to approximately 80-90 Mbit/s for FreeBSD clients and to approximately 40 Mbit/s for Windows clients.
I did try to play with kernel options
and OpenVPN options
but this did not help.
I'd appreciate any hint on what I might be doing wrong. Thanks!
I have a fairly standard bridged security/openvpn server listening on vmx3f0 on port UDP/1194. The traffic leaves on the tap0 interface which is bridged to the vmx3f1 interface.
Code:
+-------FreeBSD 9.1 on ESXi 5.0----------+
| |
| +-------+ | +-+
| +--+bridge0+--+ | |v|
| | +-------+ | | |S|
| | | | |w|
+------+ +-+--+ +--+---+ |i|
|vmx3f0+==#openvpn#--+tap0| |vmx3f1+----+t|
+------+ +----+ +------+ |c|
| | |h|
+----------------------------------------+ +-+
# ifconfig bridge0 create
# ifconfig tap0 create
# ifconfig bridge0 addm tap0 addm vmx3f1 up
# ifconfig vmx3f1 up
# ifconfig bridge0
Code:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:92:09:21:cc:00
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vmx3f1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 31250000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
Since the tap0 interface is down, we make it go up when OpenVPN opens it.
# sysctl net.link.tap.up_on_open=1 # default 0
Filtering is disabled on bridges.
# sysctl net.link.bridge.pfil_member=0 # default 1
# sysctl net.link.bridge.pfil_bridge=0 # default 1
OpenVPN is configured and started this way with
# openvpn --config openvpn.conf
where# cat openvpn.conf
Code:
server-bridge 192.0.2.49 255.255.255.240 192.0.2.59 192.0.2.62
local 192.0.2.126
lport 1194
proto udp
persist-tun
dev tap0
keepalive 5 10
cd /usr/local/etc/openvpn
script-security 3
suppress-timestamps
verb 4
push "dhcp-option DNS 198.51.100.10"
push "dhcp-option WINS 198.51.100.10"
push "dhcp-option DNS 198.51.100.18"
push "dhcp-option WINS 198.51.100.18"
# routes
push "route 198.51.100.74 255.255.255.255 192.0.2.49"
push "route 198.51.100.76 255.255.255.255 192.0.2.49"
auth-user-pass-verify "scripts/krb-authenticate.sh" via-env
client-cert-not-required
username-as-common-name
cipher none
tls-server
tls-cipher TLS-RSA-WITH-AES-128-CBC-SHA
ca /usr/local/etc/openvpn/ssl/certs/ca_authorities.crt
dh /usr/local/etc/openvpn/ssl/keys/openvpn-dh.pem
cert /usr/local/etc/openvpn/ssl/certs/openvpn-tls.crt
key /usr/local/etc/openvpn/ssl/keys/openvpn-tls.key
persist-key
When an OpenVPN client connects, it is assigned IP address 192.0.2.59 but its MAC address is not added to the list of addresses known to bridge0:
# ifconfig bridge0 addr | grep tap0
Code:
# (empty)
If I arping() the client from a different host connected to the vSwitch
arping 192.0.2.59
, the OpenVPN client's MAC address appears in the bridge0's list:# ifconfig bridge0 addr | grep tap0
Code:
00:bd:c8:e1:b9:00 Vlan1 tap0 1199 flags=0<>
The problem is however, the address is forgotten as soon as I start ping()ing the gateway from the OpenVPN client
ping 192.0.2.49
.# ifconfig bridge0 addr | grep tap0
Code:
# (empty)
I tried a (not too nice) workaround: do not let bridge0 forget the MAC addresses learned on tap0 by making it sticky.
# ifconfig bridge0 sticky tap0
The OpenVPN client is now fully functional, but there are major performance problems. If the client communicates with hosts on the same L2 network/vSwitch (i.e. 192.0.2.48/28), the bandwidth of the tunnel approaches 400 Mbit/s. However, when the client tries to communicate with a host on L3 (IP layer) meaning it has to use the default gateway 192.2.0.49 behind the tunnel, the traffic drops to approximately 80-90 Mbit/s for FreeBSD clients and to approximately 40 Mbit/s for Windows clients.
I did try to play with kernel options
# sysctl kern.ipc.maxsockbuf=16777216 # default 2097152
# sysctl net.inet.tcp.sendbuf_max=16777216 # default 2097152
# sysctl net.inet.tcp.recvbuf_max=16777216 # default 2097152
# sysctl net.inet.udp.recvspace=262144 # default 42080
and OpenVPN options
Code:
sndbuf 8388608 # default 65536
rcvbuf 8388608 # default 65536
fragment 1400
mssfix
I'd appreciate any hint on what I might be doing wrong. Thanks!