Decentralized VPN? Is it possible? Can it be done with OpenVPN?

gnoma

Active Member

Reaction score: 19
Messages: 196

Hello,

I have many servers on many different locations. They are communicating between each other via Internet.
However there's only one site that has OpenVPN server, the rest are clients.
If this site is down then all of the clients lose connection between each other and the entire solution becomes useless.

I would appreciate if anybody can help with idea how to make a distributed/decentralized VPN.
I have experience mostly with OpenVPN and would like to use it if possible in the future, if not than any other solution is welcome ;)

Thanks
 

leebrown66

Well-Known Member

Reaction score: 142
Messages: 421

If you want to do what I think you do, it's a mixture of VPN's in tunneling mode, GRE and BGP to manage the routing.
I've done this with StrongSwan, five sites at four locations, multiple providers at two of the sites.

Construct a mesh of VPN's in tunnel mode (I have only used StrongSwan).
Within that run a GRE tunnel, so you can pass all IP addresses, effectively bypassing the IPSec conditions.
Each site runs BGP+Quagga to push out the routes to all of the connected nodes (making a BGP mesh).

If one provider goes out, routing changes to the shortest path. If one site goes out, it's removed from the routing table and falls into the default null route.

Sorry I don't have the time to explain in depth.
 
OP
G

gnoma

Active Member

Reaction score: 19
Messages: 196

I was imagining that it will not require routing - one single VPN network.
For the non-vpn (pure internet) traffic every server on every site should have it's default gateway set by it's own ISP.
 
OP
G

gnoma

Active Member

Reaction score: 19
Messages: 196

You may want t try this: https://github.com/mehrdadrad/radvpn

Caveat Emptor : I haven't try Radvpn yet but I am planning to do in the future.
Looks like exactly what I need and seems the last updates are very recent. However I haven't heard about this solution and I'm not sure how reliable and how serious project it is.
I will try to find some more information about it and probably compare it with scenario mixed between MPLS and OpenVPN.

Thanks
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,778
Messages: 33,056

I have many servers on many different locations. They are communicating between each other via Internet.
However there's only one site that has OpenVPN server, the rest are clients.
If this site is down then all of the clients lose connection between each other and the entire solution becomes useless.
A typical solution is to use IPSec to tie each location together. You can use a "full mesh" design where each location has tunnels to every other location. Or a ring design where each location is connected to two neighbors, if one node drops off traffic can be routed the other way around the ring. Which one you should use depends on a number of factors, like bandwidth and latency on each location and the number of nodes (imagine a full mesh with dozens of nodes for example).

Routing between the locations can be done "manually", where you define several static routes and modify them manually whenever there's a problem. Or automated by using BGP or other routing protocols.
 

ctaranotte

Active Member

Reaction score: 25
Messages: 134

On reflection, I would suggest using Wireguard as it is actually a peer-to-peer protocol.
You can start connection with several peers on the same interface and assign static IPs within your VPN.

Does it scale? I don't know yet but I will.
 

dch

Active Member

Reaction score: 50
Messages: 126

I've been using net/zerotier for several years and it does exactly what you want (mesh setup) with minimal setup already. TLDR set up a free account at https://my.zerotier.com/ then install the port, create a private network, join nodes. Bear in mind that udp/9993 needs to get out and in, so adjust firewalls accordingly. I've never used a simpler to get started vpn. It's not perfect, none are, but its well worth trying.

My preferred config is to assign all devices an IPv6 "6PLANE" only - no IPv4, then add the IPv6 AAAA records into public DNS. Then you can refer to server.zt.your.domain and everything Just Works.

The only 2 caveats are:

- to maintain the mesh there is a constant low level of traffic. This may matter depending on whether you have mobile devices on vpn all the time
- if you're going *out* a firewall and have many clients behind it, things can get cramped and some crappy routers fall over, or fail to maintain state tables appropriately. There are a few tweaks you can put in /var/db/zerotier-one/local.conf to adjust this:

Code:
{
    "physical": {
        "127.0.0.0/8": {
            "blacklist": true
        },
        "100.64.0.0/16": {
            "blacklist": true
        },
        "10.0.0.0/8": {
            "blacklist": true
        }
    },
    "settings": {
        "primaryPort": 0,
        "allowTcpFallbackRelay": true
    }
}
Which should mostly be obvious - you can tell ZT not to send traffic out those interfaces, and the PrimaryPort one I use on stuff likely to be behind NATs (laptops, phones etc) which just says "let the OS pick my port", instead of 9993. This means that (say at home) with several devices behind a NAT, they're not all using 9993 and getting each other confused.

AMA, I'm the port maintainer.
 
OP
G

gnoma

Active Member

Reaction score: 19
Messages: 196

Hello dch,

net/zerotier sounds like exactly what I needed. I was very interested in this solution and even created 2 VMs for test.
However before starting the setup I wanted to read the documentation https://www.zerotier.com/manual/. I noticed that there's something called "planet earth" which is a public service provided by 3rd party vendor.
Seems like there's no full self hosting option. Yes, you can create "moons" that will serve as local nodes for low latency proxies so you don't have to reach earth all the time. Also a moon protects you from internet connectivity issues so you can operate without internet in your datacenter for a while. But at the end a complete setup is not possible without using the planet earth as a service or build/host your own planet earth.

I know administrators these days have no concern using such 3rd party or cloud services and probably for most of the people here it would be OK. However I'm a little conservative and prefer to rely on my own hosting for such important things as a layer2 networking, VPN and encryption between replicating databases. I realize that cloud service may provide better reliability than my own systems but still I prefer to use my own stuff ;)

So I will search a little more for net/zerotier and possibility of fully self-hosted network and if I don't find it as available option I will move on to another solution.

If I couldn't host myself a VPN probably net/zerotier would be my choice. In parallel I will start implementing and testing IPSec VPN acting like MPLS between 2 data centers and then OpenVPN acting as 2 redundant OpenVPN servers providing access to the same VPN network. Then all other data centers will only be VPN clients. It's not fully decentralized but I prefer having everything in one single network than having multiple networks and playing around with routings and OSPF on top of the complicated enough VPN setup.
 

dch

Active Member

Reaction score: 50
Messages: 126

I think if you set up zerotier with your own moons then you can skip "earth", and by disabling TCP fallback there shouldn't be a situation where you /need/ their infrastructure.

The thing I like the most about ZT is that it's a mesh so I don't need to dog-leg across the VPN if the host is nearby from a network perspective. I believe ZT 2.0 will make it much easier to run your own fully separate environments, but if you're absolutely certain you want to be separate today, best to modify the source to avoid publically available root servers.

See https://zerotier.com/zerotier-root-server-upgrades/ for the background and look in https://github.com/zerotier/ZeroTie...f2d0ac52046edd6cdbf/node/Topology.cpp#L25-L26 for example they updated theirs in https://github.com/zerotier/ZeroTierOne/commit/ecee3 if you think there's a generic patch that would enable this, I'm happy to roll it into the port for FreeBSD users, but I think ZT won't work without some moons or root servers available.
 

rf10

Member

Reaction score: 1
Messages: 20

A typical solution is to use IPSec to tie each location together. You can use a "full mesh" design where each location has tunnels to every other location. Or a ring design where each location is connected to two neighbors, if one node drops off traffic can be routed the other way around the ring. Which one you should use depends on a number of factors, like bandwidth and latency on each location and the number of nodes (imagine a full mesh with dozens of nodes for example).

Routing between the locations can be done "manually", where you define several static routes and modify them manually whenever there's a problem. Or automated by using BGP or other routing protocols.
You can do the same thing with OpenVPN. The OP posted that they use a single OpenVPN Server with several clients. There is no reason why each location cannot have its own server and clients connecting to one or more other nodes (in whatever configuration you want - mesh, ring, etc.). There is OpenVPN Cookbook which is reasonably complete with examples.

You would still need to solve the routing problem, as SirDice pointed out. As the number of sites grow, "manual" configuration may be not very scalable.
 

LVLouisCyphre

Member

Reaction score: 13
Messages: 72

You would still need to solve the routing problem, as SirDice pointed out. As the number of sites grow, "manual" configuration may be not very scalable.
Coming from supporting a very large WAN using BGP, EIGRP and OSPF, I see migraines on the split horizon. What the OP appears to be looking for dynamic multipoint VPN. I don't know if the other solutions mentioned above provide the same functionality.

It appears to get this to function with FreeBSD, the OP needs run OpenNHRP to implement dynamic multipoint VPN and run an OpenVPN server at each site. We don't have an OpenNHRP port yet. This might be feasible under Linux emulation. This also might be something that someone wishes to tackle as a port.

I'm very interested on the resolution of this issue particularly if it spawns an OpenNHRP port. I can see this being very useful; a very well welcomed addition to all BSDs along with OPN and pf Sense.
 
OP
G

gnoma

Active Member

Reaction score: 19
Messages: 196

Just to let you know the solution I have chosen, tested and since today working almost in production is security/tinc.
It's exactly what I described above and what I needed.

My setup is tinc configured in switch mode with 5 sites which resulted into a single layer2 network with 5 hosts in.
Every node is configured to reach the addresses of all other nodes - mesh setup.

There are many configuration examples on the Internet, still if anybody needs sample config please let me know.
 

LVLouisCyphre

Member

Reaction score: 13
Messages: 72

How are you dealing with next hops if a site goes down? Did you get OpenNHRP to work under Linux emulation? I'm very interested to know how you handled this issue.
 
OP
G

gnoma

Active Member

Reaction score: 19
Messages: 196

LVLouisCyphre, not sure what are you talking about.

For the public Internet connectivity (VPN endpoints) I depend on the Internet itself - every site has it's own ISP.
Inside the VPN network there's no hops. It's a single layer2 network with no routing. Single segment and single broadcast domain. It uses /dev/tap interface just like OpenVPN.
I see every other site with it's MAC address and IP address on my arp table.
If a site goes down I just stop seeing it's MAC/IP address and on the live sites I got "host down" error ;)

Does this answer your question or some how I didn't understand it right?
 
Top