OpenVPN poor performance

J

jbo@

Developer
Let there be machines A and B.:
  • A is a baremetal server in a datacenter with 1Gbps connection to the internet. 4x 3.4 GHz Xeon with 32 GB of RAM
  • B is a virtual machine running on a KVM cluster in a datacenter with 1Gbps connection to the internet. 4x 2.6 GHz KVM with 8 GB of RAM
Both machines run FreeBSD. I am using OpenVPN to tie them together. I'm using iperf3(1) for performance testing. Here are the results using the public IPs of the servers:
Code: 
root@A:~ # iperf3 -c 1.2.3.4
Connecting to host 1.2.3.4, port 5201
[  5] local 5.6.7.8 port 18305 connected to 1.2.3.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.98 MBytes  33.4 Mbits/sec    0    239 KBytes
[  5]   1.00-2.00   sec  11.4 MBytes  95.6 Mbits/sec    0    478 KBytes
[  5]   2.00-3.00   sec  19.9 MBytes   167 Mbits/sec    0    751 KBytes
[  5]   3.00-4.00   sec  28.7 MBytes   241 Mbits/sec    0   1019 KBytes
[  5]   4.00-5.00   sec  37.7 MBytes   316 Mbits/sec    0   1.25 MBytes
[  5]   5.00-6.00   sec  45.8 MBytes   384 Mbits/sec    0   1.51 MBytes
[  5]   6.00-7.00   sec  53.9 MBytes   452 Mbits/sec    0   1.76 MBytes
[  5]   7.00-8.00   sec  57.8 MBytes   485 Mbits/sec    0   1.76 MBytes
[  5]   8.00-8.63   sec  36.2 MBytes   482 Mbits/sec    0   1.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-8.63   sec   295 MBytes   287 Mbits/sec    0             sender
[  5]   0.00-8.63   sec  0.00 Bytes  0.00 bits/sec                  receiver

And here's the same going through OpenVPN:
Code: 
root@A:~ # iperf3 -c 10.8.0.46
Connecting to host 10.8.0.46, port 5201
[  5] local 10.8.0.1 port 37447 connected to 10.8.0.46 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.45 MBytes  12.1 Mbits/sec    0   1.33 KBytes
[  5]   1.00-2.00   sec  1.53 MBytes  12.8 Mbits/sec    2   39.7 KBytes
[  5]   2.00-3.00   sec  1.56 MBytes  13.1 Mbits/sec    1   47.6 KBytes
[  5]   3.00-4.00   sec  1.55 MBytes  13.0 Mbits/sec    1   53.0 KBytes
[  5]   4.00-5.00   sec  1.64 MBytes  13.8 Mbits/sec    1   60.9 KBytes
[  5]   5.00-6.00   sec  1.60 MBytes  13.5 Mbits/sec    1   62.3 KBytes
[  5]   6.00-7.00   sec  1.66 MBytes  13.9 Mbits/sec    2   35.7 KBytes
[  5]   7.00-8.00   sec  1.50 MBytes  12.5 Mbits/sec    1   42.3 KBytes
[  5]   8.00-9.00   sec  1.58 MBytes  13.2 Mbits/sec    1   49.0 KBytes
[  5]   9.00-10.00  sec  1.56 MBytes  13.1 Mbits/sec    1   55.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  15.6 MBytes  13.1 Mbits/sec   11             sender
[  5]   0.00-10.03  sec  15.5 MBytes  13.0 Mbits/sec                  receiver

I fully understand that OpenVPN will come at a performance loss. However, I would have never expected this much!
I did a lot of googling but I couldn't find anything wrong with my OpenVPN config. Here's the one of the server (host A):
Code: 
port 1194
proto udp
dev tun
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/openvpn-server.crt
key /usr/local/etc/openvpn/keys/openvpn-server.key  # This file should be kept secret
dh /usr/local/etc/openvpn/keys/dh.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
client-to-client
keepalive 10 120
tls-auth /usr/local/etc/openvpn/keys/ta.key 0 # This file is secret
cipher AES-256-CBC
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append  openvpn.log
verb 3
explicit-exit-notify 0
 
# Trying to improve performance...
sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"
And here's the one of the client (host B):
Code: 
client
dev tun
proto udp
remote my.fancy.openvpn.host.com 1194
remote-random
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
askpass /usr/local/etc/openvpn/cert.passphrase
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/openvpn-client.crt
key /usr/local/etc/openvpn/keys/openvpn-client.key
remote-cert-tls server
tls-auth /usr/local/etc/openvpn/keys/ta.key 1
cipher AES-256-CBC
verb 3
As you can see I started dicking around with the sndbuf and rcvbuf directives but they didn't have any noticeable impact.

I have also checked the system usage on both hosts. Neither of the hosts seem to have insufficient hardware performance.

I'd appreciate any kind of help!
 
I set cipher to none on both the server and the client. Then I restarted OpenVPN on both using service openvpn restart. I am getting very similar results (although less stable):
Code: 
root@hydrogen1:/usr/local/etc/openvpn # iperf3 -c 10.8.0.46
Connecting to host 10.8.0.46, port 5201
[  5] local 10.8.0.1 port 31770 connected to 10.8.0.46 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.31 MBytes  11.0 Mbits/sec    1   36.9 KBytes
[  5]   1.00-2.00   sec  1.20 MBytes  10.1 Mbits/sec    1   51.5 KBytes
[  5]   2.00-3.00   sec  1.12 MBytes  9.37 Mbits/sec    3   22.4 KBytes
[  5]   3.00-4.00   sec  1.04 MBytes  8.73 Mbits/sec    6   39.6 KBytes
[  5]   4.00-5.00   sec  1.23 MBytes  10.3 Mbits/sec    6   54.2 KBytes
[  5]   5.00-6.00   sec   824 KBytes  6.74 Mbits/sec    3   30.4 KBytes
[  5]   6.00-7.00   sec  1.22 MBytes  10.2 Mbits/sec    1   46.2 KBytes
[  5]   7.00-8.00   sec  1.34 MBytes  11.2 Mbits/sec    1   60.8 KBytes
[  5]   8.00-9.00   sec  1.54 MBytes  12.9 Mbits/sec    3   35.5 KBytes
[  5]   9.00-10.00  sec  1.47 MBytes  12.4 Mbits/sec    1   41.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  12.3 MBytes  10.3 Mbits/sec   26             sender
[  5]   0.00-10.03  sec  12.2 MBytes  10.2 Mbits/sec                  receiver

iperf Done.

I also played around with tun-mtu, mssfix and fragment. I didn't notice any changes there either. What am I missing? Any other ideas?
 
I did some more testing: I used the -P option of iperf3(1) to run 20 client streams in parallel. If I understand correctly, they are indeed running truly in parallel and not sequentially. I am getting much better results:
Code: 
root@hydrogen1:~ # iperf3 -c 10.8.0.46 -P 20
Connecting to host 10.8.0.46, port 5201
[  5] local 10.8.0.1 port 57475 connected to 10.8.0.46 port 5201
[  7] local 10.8.0.1 port 57476 connected to 10.8.0.46 port 5201
[  9] local 10.8.0.1 port 57483 connected to 10.8.0.46 port 5201
[ 11] local 10.8.0.1 port 57484 connected to 10.8.0.46 port 5201
[ 13] local 10.8.0.1 port 57487 connected to 10.8.0.46 port 5201
[ 15] local 10.8.0.1 port 57492 connected to 10.8.0.46 port 5201
[ 17] local 10.8.0.1 port 57498 connected to 10.8.0.46 port 5201
[ 19] local 10.8.0.1 port 57513 connected to 10.8.0.46 port 5201
[ 21] local 10.8.0.1 port 57521 connected to 10.8.0.46 port 5201
[ 23] local 10.8.0.1 port 57532 connected to 10.8.0.46 port 5201
[ 25] local 10.8.0.1 port 57533 connected to 10.8.0.46 port 5201
[ 27] local 10.8.0.1 port 57534 connected to 10.8.0.46 port 5201
[ 29] local 10.8.0.1 port 57549 connected to 10.8.0.46 port 5201
[ 31] local 10.8.0.1 port 57550 connected to 10.8.0.46 port 5201
[ 33] local 10.8.0.1 port 57554 connected to 10.8.0.46 port 5201
[ 35] local 10.8.0.1 port 57556 connected to 10.8.0.46 port 5201
[ 37] local 10.8.0.1 port 57557 connected to 10.8.0.46 port 5201
[ 39] local 10.8.0.1 port 57564 connected to 10.8.0.46 port 5201
[ 41] local 10.8.0.1 port 57566 connected to 10.8.0.46 port 5201
[ 43] local 10.8.0.1 port 57568 connected to 10.8.0.46 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   662 KBytes  5.43 Mbits/sec    8   17.1 KBytes
[  7]   0.00-1.00   sec   629 KBytes  5.15 Mbits/sec    0   38.7 KBytes
[  9]   0.00-1.00   sec   628 KBytes  5.14 Mbits/sec    0   38.7 KBytes
[ 11]   0.00-1.00   sec   629 KBytes  5.15 Mbits/sec    0   38.7 KBytes
[ 13]   0.00-1.00   sec   628 KBytes  5.14 Mbits/sec    0   38.7 KBytes
[ 15]   0.00-1.00   sec   636 KBytes  5.21 Mbits/sec    0   38.7 KBytes
[ 17]   0.00-1.00   sec   628 KBytes  5.14 Mbits/sec    0   38.7 KBytes
[ 19]   0.00-1.00   sec   485 KBytes  3.97 Mbits/sec    3   11.9 KBytes
[ 21]   0.00-1.00   sec  85.7 KBytes   702 Kbits/sec    4   9.25 KBytes
[ 23]   0.00-1.00   sec   474 KBytes  3.88 Mbits/sec    3   31.8 KBytes
[ 25]   0.00-1.00   sec  85.7 KBytes   702 Kbits/sec    4   9.25 KBytes
[ 27]   0.00-1.00   sec   469 KBytes  3.84 Mbits/sec    3   31.8 KBytes
[ 29]   0.00-1.00   sec   329 KBytes  2.70 Mbits/sec    6   9.24 KBytes
[ 31]   0.00-1.00   sec   432 KBytes  3.54 Mbits/sec    5   11.9 KBytes
[ 33]   0.00-1.00   sec   437 KBytes  3.58 Mbits/sec    4   11.9 KBytes
[ 35]   0.00-1.00   sec   561 KBytes  4.59 Mbits/sec    2   35.7 KBytes
[ 37]   0.00-1.00   sec   561 KBytes  4.59 Mbits/sec    2   35.7 KBytes
[ 39]   0.00-1.00   sec   561 KBytes  4.59 Mbits/sec    2   35.7 KBytes
[ 41]   0.00-1.00   sec   582 KBytes  4.77 Mbits/sec    2   37.1 KBytes
[ 43]   0.00-1.00   sec   496 KBytes  4.07 Mbits/sec    3   34.4 KBytes
[SUM]   0.00-1.00   sec  9.76 MBytes  81.9 Mbits/sec   51
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  1.42 MBytes  11.9 Mbits/sec    5   34.2 KBytes
[  7]   1.00-2.00   sec  56.6 KBytes   464 Kbits/sec   40   9.26 KBytes
[  9]   1.00-2.00   sec   456 KBytes  3.74 Mbits/sec   42   18.5 KBytes
[ 11]   1.00-2.00   sec   563 KBytes  4.61 Mbits/sec   45   19.7 KBytes
[ 13]   1.00-2.00   sec   622 KBytes  5.10 Mbits/sec   39   43.7 KBytes
[ 15]   1.00-2.00   sec   501 KBytes  4.10 Mbits/sec   41   19.8 KBytes
[ 17]   1.00-2.00   sec   622 KBytes  5.10 Mbits/sec   39   43.7 KBytes
[ 19]   1.00-2.00   sec   170 KBytes  1.39 Mbits/sec   30   6.63 KBytes
[ 21]   1.00-2.00   sec  1.05 MBytes  8.77 Mbits/sec    5   26.3 KBytes
[ 23]   1.00-2.00   sec   683 KBytes  5.59 Mbits/sec    5   27.8 KBytes
[ 25]   1.00-2.00   sec  1.08 MBytes  9.10 Mbits/sec    0   53.0 KBytes
[ 27]   1.00-2.00   sec   843 KBytes  6.90 Mbits/sec    3   37.1 KBytes
[ 29]   1.00-2.00   sec   613 KBytes  5.02 Mbits/sec    6   1.33 KBytes
[ 31]   1.00-2.00   sec   643 KBytes  5.27 Mbits/sec   11   1.33 KBytes
[ 33]   1.00-2.00   sec  1.07 MBytes  8.97 Mbits/sec    9   1.33 KBytes
[ 35]   1.00-2.00   sec  2.03 MBytes  17.0 Mbits/sec    0   80.8 KBytes
[ 37]   1.00-2.00   sec   971 KBytes  7.95 Mbits/sec    3   33.1 KBytes
[ 39]   1.00-2.00   sec   884 KBytes  7.24 Mbits/sec    4   35.8 KBytes
[ 41]   1.00-2.00   sec  1.14 MBytes  9.59 Mbits/sec    6   40.9 KBytes
[ 43]   1.00-2.00   sec  1.16 MBytes  9.70 Mbits/sec    1   35.8 KBytes
[SUM]   1.00-2.00   sec  16.4 MBytes   137 Mbits/sec  334
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  1.25 MBytes  10.5 Mbits/sec    7   54.2 KBytes
[  7]   2.00-3.00   sec  1.08 MBytes  9.07 Mbits/sec    0   54.3 KBytes
[  9]   2.00-3.00   sec  1.04 MBytes  8.74 Mbits/sec    2   51.6 KBytes
[ 11]   2.00-3.00   sec   978 KBytes  8.01 Mbits/sec    4   44.9 KBytes
[ 13]   2.00-3.00   sec  2.28 MBytes  19.1 Mbits/sec    0   88.8 KBytes
[ 15]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec    2   54.4 KBytes
[ 17]   2.00-3.00   sec  2.21 MBytes  18.6 Mbits/sec    0   87.5 KBytes
[ 19]   2.00-3.00   sec  1014 KBytes  8.31 Mbits/sec    1   51.8 KBytes
[ 21]   2.00-3.00   sec  1.11 MBytes  9.29 Mbits/sec    6   48.9 KBytes
[ 23]   2.00-3.00   sec  1000 KBytes  8.20 Mbits/sec    6   38.5 KBytes
[ 25]   2.00-3.00   sec  2.61 MBytes  21.9 Mbits/sec    0   98.1 KBytes
[ 27]   2.00-3.00   sec  1.01 MBytes  8.48 Mbits/sec    4   42.3 KBytes
[ 29]   2.00-3.00   sec   406 KBytes  3.33 Mbits/sec   16   6.57 KBytes
[ 31]   2.00-3.00   sec   476 KBytes  3.90 Mbits/sec    6   34.4 KBytes
[ 33]   2.00-3.00   sec  1.02 MBytes  8.60 Mbits/sec   40   56.8 KBytes
[ 35]   2.00-3.00   sec  2.39 MBytes  20.0 Mbits/sec    2   34.5 KBytes
[ 37]   2.00-3.00   sec   709 KBytes  5.81 Mbits/sec   11   29.2 KBytes
[ 39]   2.00-3.00   sec   553 KBytes  4.53 Mbits/sec   11   22.5 KBytes
[ 41]   2.00-3.00   sec  1.42 MBytes  11.9 Mbits/sec   18   22.3 KBytes
[ 43]   2.00-3.00   sec   653 KBytes  5.35 Mbits/sec   13   26.6 KBytes
[SUM]   2.00-3.00   sec  24.2 MBytes   203 Mbits/sec  149
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   650 KBytes  5.32 Mbits/sec   68   13.3 KBytes
[  7]   3.00-4.00   sec   498 KBytes  4.08 Mbits/sec   65   35.7 KBytes
[  9]   3.00-4.00   sec   381 KBytes  3.12 Mbits/sec   54   38.4 KBytes
[ 11]   3.00-4.00   sec  1.13 MBytes  9.48 Mbits/sec   20   16.0 KBytes
[ 13]   3.00-4.00   sec   281 KBytes  2.30 Mbits/sec  125   17.2 KBytes
[ 15]   3.00-4.00   sec   417 KBytes  3.41 Mbits/sec   84   7.96 KBytes
[ 17]   3.00-4.00   sec  1.80 MBytes  15.1 Mbits/sec   92   9.20 KBytes
[ 19]   3.00-4.00   sec   420 KBytes  3.44 Mbits/sec   65   7.92 KBytes
[ 21]   3.00-4.00   sec  1.41 MBytes  11.9 Mbits/sec   64   28.8 KBytes
[ 23]   3.00-4.00   sec   401 KBytes  3.28 Mbits/sec   38   14.5 KBytes
[ 25]   3.00-4.00   sec  1.83 MBytes  15.4 Mbits/sec   38   46.1 KBytes
[ 27]   3.00-4.00   sec   392 KBytes  3.21 Mbits/sec   54   15.8 KBytes
[ 29]   3.00-4.00   sec   705 KBytes  5.77 Mbits/sec   22   23.7 KBytes
[ 31]   3.00-4.00   sec  93.4 KBytes   765 Kbits/sec   59   5.26 KBytes
[ 33]   3.00-4.00   sec   388 KBytes  3.18 Mbits/sec   44   29.1 KBytes
[ 35]   3.00-4.00   sec   825 KBytes  6.76 Mbits/sec   19   14.5 KBytes
[ 37]   3.00-4.00   sec   808 KBytes  6.62 Mbits/sec   13   45.0 KBytes
[ 39]   3.00-4.00   sec  60.5 KBytes   496 Kbits/sec   29   3.96 KBytes
[ 41]   3.00-4.00   sec   848 KBytes  6.95 Mbits/sec   48   14.8 KBytes
[ 43]   3.00-4.00   sec   354 KBytes  2.90 Mbits/sec   19   33.0 KBytes
[SUM]   3.00-4.00   sec  13.5 MBytes   113 Mbits/sec  1020
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   577 KBytes  4.73 Mbits/sec   19   25.1 KBytes
[  7]   4.00-5.00   sec  1.44 MBytes  12.1 Mbits/sec    6   51.5 KBytes
[  9]   4.00-5.00   sec  1.35 MBytes  11.3 Mbits/sec    8   58.1 KBytes
[ 11]   4.00-5.00   sec   228 KBytes  1.86 Mbits/sec   32   5.26 KBytes
[ 13]   4.00-5.00   sec   941 KBytes  7.71 Mbits/sec   39   14.5 KBytes
[ 15]   4.00-5.00   sec   316 KBytes  2.59 Mbits/sec   41   12.1 KBytes
[ 17]   4.00-5.00   sec   277 KBytes  2.27 Mbits/sec   22   13.2 KBytes
[ 19]   4.00-5.00   sec   502 KBytes  4.11 Mbits/sec    8   35.9 KBytes
[ 21]   4.00-5.00   sec   509 KBytes  4.17 Mbits/sec   55   7.90 KBytes
[ 23]   4.00-5.00   sec   540 KBytes  4.43 Mbits/sec   24   15.9 KBytes
[ 25]   4.00-5.00   sec  1.40 MBytes  11.8 Mbits/sec   15   10.5 KBytes
[ 27]   4.00-5.00   sec   706 KBytes  5.78 Mbits/sec   15   23.8 KBytes
[ 29]   4.00-5.00   sec   402 KBytes  3.30 Mbits/sec   20   5.27 KBytes
[ 31]   4.00-5.00   sec   222 KBytes  1.82 Mbits/sec   17   13.1 KBytes
[ 33]   4.00-5.00   sec   581 KBytes  4.76 Mbits/sec   39   35.7 KBytes
[ 35]   4.00-5.00   sec   233 KBytes  1.91 Mbits/sec   15   13.2 KBytes
[ 37]   4.00-5.00   sec   818 KBytes  6.70 Mbits/sec   27   25.0 KBytes
[ 39]   4.00-5.00   sec   492 KBytes  4.03 Mbits/sec    5   29.1 KBytes
[ 41]   4.00-5.00   sec   819 KBytes  6.71 Mbits/sec   13   19.7 KBytes
[ 43]   4.00-5.00   sec   741 KBytes  6.07 Mbits/sec   26   13.2 KBytes
[SUM]   4.00-5.00   sec  12.9 MBytes   108 Mbits/sec  446
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  1006 KBytes  8.24 Mbits/sec   25   22.6 KBytes
[  7]   5.00-6.00   sec   723 KBytes  5.92 Mbits/sec   45   30.4 KBytes
[  9]   5.00-6.00   sec  1.85 MBytes  15.5 Mbits/sec   18   34.2 KBytes
[ 11]   5.00-6.00   sec   447 KBytes  3.66 Mbits/sec   37   11.8 KBytes
[ 13]   5.00-6.00   sec   534 KBytes  4.37 Mbits/sec   23   26.6 KBytes
[ 15]   5.00-6.00   sec   193 KBytes  1.58 Mbits/sec    9   22.5 KBytes
[ 17]   5.00-6.00   sec  1.11 MBytes  9.33 Mbits/sec   15   17.1 KBytes
[ 19]   5.00-6.00   sec  1.27 MBytes  10.6 Mbits/sec   24   10.5 KBytes
[ 21]   5.00-6.00   sec   160 KBytes  1.31 Mbits/sec    8   13.2 KBytes
[ 23]   5.00-6.00   sec   819 KBytes  6.71 Mbits/sec   17   21.0 KBytes
[ 25]   5.00-6.00   sec  1.13 MBytes  9.46 Mbits/sec   71   20.1 KBytes
[ 27]   5.00-6.00   sec   390 KBytes  3.20 Mbits/sec   23   29.0 KBytes
[ 29]   5.00-6.00   sec   268 KBytes  2.20 Mbits/sec   15   21.2 KBytes
[ 31]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec    0   58.2 KBytes
[ 33]   5.00-6.00   sec   405 KBytes  3.32 Mbits/sec   50   17.2 KBytes
[ 35]   5.00-6.00   sec   740 KBytes  6.06 Mbits/sec    9   45.0 KBytes
[ 37]   5.00-6.00   sec   445 KBytes  3.64 Mbits/sec   41   23.8 KBytes
[ 39]   5.00-6.00   sec   950 KBytes  7.79 Mbits/sec   23   11.8 KBytes
[ 41]   5.00-6.00   sec   368 KBytes  3.02 Mbits/sec   34   21.2 KBytes
[ 43]   5.00-6.00   sec   287 KBytes  2.35 Mbits/sec   21   17.2 KBytes
[SUM]   5.00-6.00   sec  14.1 MBytes   118 Mbits/sec  508
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   824 KBytes  6.75 Mbits/sec   12   2.63 KBytes
[  7]   6.00-7.00   sec  1.51 MBytes  12.6 Mbits/sec   38   9.20 KBytes
[  9]   6.00-7.00   sec  1.30 MBytes  10.9 Mbits/sec    7   1.31 KBytes
[ 11]   6.00-7.00   sec   505 KBytes  4.13 Mbits/sec   17   9.20 KBytes
[ 13]   6.00-7.00   sec   122 KBytes  1.00 Mbits/sec   35   14.6 KBytes
[ 15]   6.00-7.00   sec  1.15 MBytes  9.67 Mbits/sec    3   42.2 KBytes
[ 17]   6.00-7.00   sec   563 KBytes  4.61 Mbits/sec   42   28.0 KBytes
[ 19]   6.00-7.00   sec   312 KBytes  2.55 Mbits/sec   18   5.26 KBytes
[ 21]   6.00-7.00   sec   540 KBytes  4.42 Mbits/sec    9   38.4 KBytes
[ 23]   6.00-7.00   sec   514 KBytes  4.21 Mbits/sec   13   15.8 KBytes
[ 25]   6.00-7.00   sec  62.1 KBytes   509 Kbits/sec   28   6.61 KBytes
[ 27]   6.00-7.00   sec  1010 KBytes  8.27 Mbits/sec   14   29.0 KBytes
[ 29]   6.00-7.00   sec  1.03 MBytes  8.60 Mbits/sec    4   21.1 KBytes
[ 31]   6.00-7.00   sec  1.10 MBytes  9.23 Mbits/sec   36   23.7 KBytes
[ 33]   6.00-7.00   sec   229 KBytes  1.87 Mbits/sec   29   17.3 KBytes
[ 35]   6.00-7.00   sec   956 KBytes  7.83 Mbits/sec    5   27.7 KBytes
[ 37]   6.00-7.00   sec   187 KBytes  1.53 Mbits/sec   31   6.61 KBytes
[ 39]   6.00-7.00   sec   216 KBytes  1.77 Mbits/sec   18   13.2 KBytes
[ 41]   6.00-7.00   sec   875 KBytes  7.16 Mbits/sec   59   18.4 KBytes
[ 43]   6.00-7.00   sec   803 KBytes  6.58 Mbits/sec   13   23.7 KBytes
[SUM]   6.00-7.00   sec  13.6 MBytes   114 Mbits/sec  431
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   268 KBytes  2.20 Mbits/sec   21   22.4 KBytes
[  7]   7.00-8.00   sec   377 KBytes  3.09 Mbits/sec   27   6.61 KBytes
[  9]   7.00-8.00   sec   184 KBytes  1.51 Mbits/sec   46   7.95 KBytes
[ 11]   7.00-8.00   sec   292 KBytes  2.39 Mbits/sec   22   2.63 KBytes
[ 13]   7.00-8.00   sec  1.21 MBytes  10.1 Mbits/sec    0   59.7 KBytes
[ 15]   7.00-8.00   sec  2.21 MBytes  18.5 Mbits/sec    0   87.3 KBytes
[ 17]   7.00-8.00   sec  1.65 MBytes  13.8 Mbits/sec    0   71.8 KBytes
[ 19]   7.00-8.00   sec   262 KBytes  2.14 Mbits/sec   11   2.63 KBytes
[ 21]   7.00-8.00   sec   233 KBytes  1.91 Mbits/sec   54   5.28 KBytes
[ 23]   7.00-8.00   sec   141 KBytes  1.15 Mbits/sec   21   3.96 KBytes
[ 25]   7.00-8.00   sec   904 KBytes  7.41 Mbits/sec    0   49.0 KBytes
[ 27]   7.00-8.00   sec   446 KBytes  3.65 Mbits/sec   26   27.8 KBytes
[ 29]   7.00-8.00   sec  1.46 MBytes  12.3 Mbits/sec    0   66.2 KBytes
[ 31]   7.00-8.00   sec   340 KBytes  2.79 Mbits/sec   21   13.2 KBytes
[ 33]   7.00-8.00   sec  1.27 MBytes  10.7 Mbits/sec    0   61.1 KBytes
[ 35]   7.00-8.00   sec   816 KBytes  6.69 Mbits/sec    5   31.7 KBytes
[ 37]   7.00-8.00   sec   450 KBytes  3.68 Mbits/sec   18   10.6 KBytes
[ 39]   7.00-8.00   sec   489 KBytes  4.01 Mbits/sec   22   10.6 KBytes
[ 41]   7.00-8.00   sec   178 KBytes  1.46 Mbits/sec   28   6.62 KBytes
[ 43]   7.00-8.00   sec   593 KBytes  4.86 Mbits/sec   35   17.3 KBytes
[SUM]   7.00-8.00   sec  13.6 MBytes   114 Mbits/sec  357
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   380 KBytes  3.11 Mbits/sec   18   4.06 KBytes
[  7]   8.00-9.00   sec   300 KBytes  2.46 Mbits/sec   27   2.64 KBytes
[  9]   8.00-9.00   sec   507 KBytes  4.16 Mbits/sec   32   17.2 KBytes
[ 11]   8.00-9.00   sec   875 KBytes  7.17 Mbits/sec   20   30.3 KBytes
[ 13]   8.00-9.00   sec   394 KBytes  3.23 Mbits/sec   48   5.28 KBytes
[ 15]   8.00-9.00   sec   806 KBytes  6.61 Mbits/sec   71   41.0 KBytes
[ 17]   8.00-9.00   sec  1.11 MBytes  9.29 Mbits/sec    9   35.6 KBytes
[ 19]   8.00-9.00   sec   548 KBytes  4.49 Mbits/sec   29   17.3 KBytes
[ 21]   8.00-9.00   sec   143 KBytes  1.17 Mbits/sec    7   15.9 KBytes
[ 23]   8.00-9.00   sec   166 KBytes  1.36 Mbits/sec   10   17.2 KBytes
[ 25]   8.00-9.00   sec  2.40 MBytes  20.1 Mbits/sec    0   94.1 KBytes
[ 27]   8.00-9.00   sec   778 KBytes  6.37 Mbits/sec   28   31.7 KBytes
[ 29]   8.00-9.00   sec   511 KBytes  4.18 Mbits/sec   79   37.0 KBytes
[ 31]   8.00-9.00   sec   651 KBytes  5.33 Mbits/sec   42   19.9 KBytes
[ 33]   8.00-9.00   sec   750 KBytes  6.14 Mbits/sec   83   14.5 KBytes
[ 35]   8.00-9.00   sec  1.22 MBytes  10.3 Mbits/sec   26   43.8 KBytes
[ 37]   8.00-9.00   sec   596 KBytes  4.88 Mbits/sec   42   28.1 KBytes
[ 39]   8.00-9.00   sec   162 KBytes  1.32 Mbits/sec   20   1.33 KBytes
[ 41]   8.00-9.00   sec   718 KBytes  5.88 Mbits/sec    5   31.7 KBytes
[ 43]   8.00-9.00   sec   377 KBytes  3.09 Mbits/sec   23   2.63 KBytes
[SUM]   8.00-9.00   sec  13.2 MBytes   111 Mbits/sec  619
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   367 KBytes  3.00 Mbits/sec    3   19.8 KBytes
[  7]   9.00-10.00  sec  93.4 KBytes   765 Kbits/sec   24   6.58 KBytes
[  9]   9.00-10.00  sec   304 KBytes  2.49 Mbits/sec   34   6.60 KBytes
[ 11]   9.00-10.00  sec   521 KBytes  4.27 Mbits/sec   27   14.5 KBytes
[ 13]   9.00-10.00  sec   894 KBytes  7.32 Mbits/sec    0   49.0 KBytes
[ 15]   9.00-10.00  sec  1.24 MBytes  10.4 Mbits/sec    1   34.5 KBytes
[ 17]   9.00-10.00  sec   506 KBytes  4.15 Mbits/sec   27   1.31 KBytes
[ 19]   9.00-10.00  sec   413 KBytes  3.38 Mbits/sec   17   31.9 KBytes
[ 21]   9.00-10.00  sec  82.8 KBytes   678 Kbits/sec   14   1.31 KBytes
[ 23]   9.00-10.00  sec  1.34 MBytes  11.2 Mbits/sec    0   62.3 KBytes
[ 25]   9.00-10.00  sec  1.29 MBytes  10.8 Mbits/sec    3   37.0 KBytes
[ 27]   9.00-10.00  sec   864 KBytes  7.08 Mbits/sec   37   33.0 KBytes
[ 29]   9.00-10.00  sec   419 KBytes  3.43 Mbits/sec   28   17.1 KBytes
[ 31]   9.00-10.00  sec   227 KBytes  1.86 Mbits/sec   23   14.5 KBytes
[ 33]   9.00-10.00  sec  1016 KBytes  8.32 Mbits/sec   24   6.57 KBytes
[ 35]   9.00-10.00  sec   787 KBytes  6.45 Mbits/sec   20   19.8 KBytes
[ 37]   9.00-10.00  sec  1.66 MBytes  13.9 Mbits/sec    0   71.9 KBytes
[ 39]   9.00-10.00  sec   572 KBytes  4.68 Mbits/sec    9   38.4 KBytes
[ 41]   9.00-10.00  sec  1.10 MBytes  9.25 Mbits/sec   41   13.1 KBytes
[ 43]   9.00-10.00  sec   177 KBytes  1.45 Mbits/sec   16   13.2 KBytes
[SUM]   9.00-10.00  sec  13.7 MBytes   115 Mbits/sec  348
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.29 MBytes  6.12 Mbits/sec  186             sender
[  5]   0.00-10.03  sec  7.18 MBytes  6.01 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  6.64 MBytes  5.57 Mbits/sec  272             sender
[  7]   0.00-10.03  sec  6.51 MBytes  5.44 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  7.95 MBytes  6.67 Mbits/sec  243             sender
[  9]   0.00-10.03  sec  7.79 MBytes  6.51 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  6.05 MBytes  5.08 Mbits/sec  224             sender
[ 11]   0.00-10.03  sec  5.92 MBytes  4.95 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  7.80 MBytes  6.54 Mbits/sec  309             sender
[ 13]   0.00-10.03  sec  7.66 MBytes  6.40 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec  8.53 MBytes  7.16 Mbits/sec  252             sender
[ 15]   0.00-10.03  sec  8.35 MBytes  6.99 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  10.4 MBytes  8.74 Mbits/sec  246             sender
[ 17]   0.00-10.03  sec  10.2 MBytes  8.56 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec  5.30 MBytes  4.44 Mbits/sec  206             sender
[ 19]   0.00-10.03  sec  5.18 MBytes  4.33 Mbits/sec                  receiver
[ 21]   0.00-10.00  sec  5.28 MBytes  4.43 Mbits/sec  226             sender
[ 21]   0.00-10.03  sec  5.13 MBytes  4.29 Mbits/sec                  receiver
[ 23]   0.00-10.00  sec  5.96 MBytes  5.00 Mbits/sec  137             sender
[ 23]   0.00-10.03  sec  5.90 MBytes  4.93 Mbits/sec                  receiver
[ 25]   0.00-10.00  sec  12.8 MBytes  10.7 Mbits/sec  159             sender
[ 25]   0.00-10.03  sec  12.6 MBytes  10.5 Mbits/sec                  receiver
[ 27]   0.00-10.00  sec  6.77 MBytes  5.68 Mbits/sec  207             sender
[ 27]   0.00-10.03  sec  6.67 MBytes  5.58 Mbits/sec                  receiver
[ 29]   0.00-10.00  sec  6.06 MBytes  5.08 Mbits/sec  196             sender
[ 29]   0.00-10.03  sec  5.93 MBytes  4.96 Mbits/sec                  receiver
[ 31]   0.00-10.00  sec  5.31 MBytes  4.45 Mbits/sec  220             sender
[ 31]   0.00-10.03  sec  5.18 MBytes  4.33 Mbits/sec                  receiver
[ 33]   0.00-10.00  sec  7.09 MBytes  5.94 Mbits/sec  322             sender
[ 33]   0.00-10.03  sec  6.98 MBytes  5.84 Mbits/sec                  receiver
[ 35]   0.00-10.00  sec  10.4 MBytes  8.76 Mbits/sec  103             sender
[ 35]   0.00-10.03  sec  10.2 MBytes  8.56 Mbits/sec                  receiver
[ 37]   0.00-10.00  sec  7.07 MBytes  5.93 Mbits/sec  188             sender
[ 37]   0.00-10.03  sec  6.99 MBytes  5.85 Mbits/sec                  receiver
[ 39]   0.00-10.00  sec  4.82 MBytes  4.05 Mbits/sec  143             sender
[ 39]   0.00-10.03  sec  4.74 MBytes  3.97 Mbits/sec                  receiver
[ 41]   0.00-10.00  sec  7.95 MBytes  6.67 Mbits/sec  254             sender
[ 41]   0.00-10.03  sec  7.88 MBytes  6.59 Mbits/sec                  receiver
[ 43]   0.00-10.00  sec  5.53 MBytes  4.64 Mbits/sec  170             sender
[ 43]   0.00-10.03  sec  5.42 MBytes  4.54 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   145 MBytes   122 Mbits/sec  4263             sender
[SUM]   0.00-10.03  sec   142 MBytes   119 Mbits/sec                  receiver

iperf Done.

While this is very synthetic, I feel like this shows that the network infrastructure and the hardware on both ends are able to keep up with more.

Any thoughts?
 
I suspect that the problem reside in your VM . You can try to install the OpenVPN on your laptop and try to run the similar test. If you are using OpenVPN only for connection between those two hosts you may want to switch to IPsec tunnel instead of OpenVPN.
 
Okay, I added a baremetal server to the VPN network and ran the same test again. I'm getting much better results:
Code: 
root@carbon:/usr/local/etc/openvpn # iperf3 -c 10.8.0.1
Connecting to host 10.8.0.1, port 5201
[  5] local 10.8.0.58 port 17684 connected to 10.8.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  16.2 MBytes   136 Mbits/sec   95   13.3 KBytes
[  5]   1.00-2.00   sec  13.2 MBytes   110 Mbits/sec  105   10.6 KBytes
[  5]   2.00-3.00   sec  12.8 MBytes   108 Mbits/sec  115   77.9 KBytes
[  5]   3.00-4.00   sec  12.1 MBytes   102 Mbits/sec  113   74.0 KBytes
[  5]   4.00-5.00   sec  12.0 MBytes   101 Mbits/sec  115   3.98 KBytes
[  5]   5.00-6.00   sec  13.2 MBytes   111 Mbits/sec  108   42.2 KBytes
[  5]   6.00-7.00   sec  22.9 MBytes   193 Mbits/sec    2   39.8 KBytes
[  5]   7.00-8.00   sec  14.2 MBytes   119 Mbits/sec    2    130 KBytes
[  5]   8.00-9.00   sec  19.2 MBytes   161 Mbits/sec    3   38.5 KBytes
[  5]   9.00-10.00  sec  14.2 MBytes   119 Mbits/sec    2    121 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   150 MBytes   126 Mbits/sec  660             sender
[  5]   0.00-10.08  sec   150 MBytes   125 Mbits/sec                  receiver
iperf Done.

This seems to indicate that my VMs are not capable of doing more. The VMs are FreeBSD running on KVM using virtio drivers. I don't expect the network interface to be the problem as the VMs are able to saturate the 1G connection using iperf3 over their public IPs. Any hints or tipps?
 
Is this without encryption?
Check your actual MTU size between the hosts without OpenVPN you can do this with sweep ping using ping -D -g 1400 -G 1550 1.2.3.4 then from the result substract 20 bytes for ip header and 8 bytes for udp header and set the value as link-mtu in OpenVPN. Do not set tun-mtu as it will be calculated based on the link-mtu. Also don't set fragment option. Then check the transfer speed without encryption and signing (--cipher none --auth none) the result should be close to the actual bandwidth between the two hosts. Then enable the encryption and test again.

If your goal is only to connect those two servers it's better to use IPsec insted of OpenVPN.
 
You must log in or register to reply here.
Back
Top