Solved Dual VPN clients connectivity issues

[dagensdorf]

Greetings colleagues,

I seem to have a curious issue (am a novice so hopefully the solution is obvious).

1. A simple home network setup with a FreeBsd 12 (release if that's important) gateway which runs a VPN and PF in a setup largely inspired by: Network Filter by Guillaume Kaddouch . In short - all internal clients are going thru the VPN connection on the gateway. There is also Dhcp, Dns and NTP servers on same machine. There are about 1/2 dozen clients on internal network. Everything has been working for a couple of years.

2. Recently the requirements have changed so that a particular internal client $FOO needs to go thru a different VPN, while the rest need to keep on going thru the original VPN.

3. So I added another openvpn directive to /etc/rc.conf like so: (following this post ). VPN providers are different.

 openvpn_all_but_FOO_enable="YES"
openvpn_all_but_FOO_configfile="path_to_original_ovpn_file"
openvpn_FOO_enable="YES"
openvpn_FOO_configfile="path_to_new_ovpn_file"
openvpn_if="tun" #as before

and verified that both VPNs are actually working.

4. I also amended the pf.conf to discriminate between $FOO and the rest of the local clients:

 nat on $VPN_FOR_ALL from <LOCAL Network except FOO>  to any -> ($VPN_FOR_ALL) #this is tun0 device, original VPN
nat on $VPN_FOR_FOO from <FOO>  to any -> ($VPN_FOR_FOO) #this is tun1 device, new VPN

I use tables in pf instead of labels so for example

 table <LOCAL Network except FOO> {$int_nic:Network, !$FOO}
table <FOO> {$FOO}

5. So the $VPN_FOR_ALL part is working (as before), but the new part for $VPN_FOR_FOO is not - there is no internet connectivity for $FOO

6. I checked the routing table and see entries for both tun0 and tun1, except tun0 has 2 extra ones:

 Destination        Gateway
0.0.0.0/1             IP for $VPN_FOR_ALL #(tun0)
128.0.0./1           IP for $VPN_FOR_ALL #(tun0)

.
.
.

So to me it looks that once tun0 is created, it adds the catch-all route to the table to make sure that everything goes thru the tunnel. And when tun1 starts, it tries the same thing but gets rejected.
Besides those 2 "extra" ones for tun0 there are other tun1 and tun0 entries in the table like so:

 a.b.c.0/24       IP for $VPN_FOR_ALL #matching the subnet for tun0
a.b.c.1            link#x                           # for tun0
a.b.c.2            link#x                           # for lo0
a.k.g.0/24       IP for $VPN_FOR_FOO #matching the subnet for tun1
a.k.g.1            link#y                           # for tun1
a.k.g.213        link#y                           # for lo0

7. I tried adjusting the .ovpn files to add/remove the pull directive to eliminate these extra routes for tun0. It worked, but there was no connectivity on either VPN that way. (if there are no such extra routes for either tun0 or tun1.

8. $VPN_FOR_ALL and $VPN_FOR_FOO subnets are different, both connections work (if I swap their order of creation and pretend to be $FOO), connectivity for all clients but $FOO works thru tun0. Both VPNs are on UDP, different ports.

So - please suggest at least the direction I should be thinking about.
Is this the routing issue? Can this be done in principle without using jails for the second VPN?

Any help/suggestions are appreciated and many thanks in advance.

Andrei

 

[dagensdorf]

Ok, I solved this, with a hack admittedly, but still it works. Maybe helpful for some of you in similar situations.
in short:
1. I used an additional routing table for the 2nd VPN and then in
/etc/pf.conf I added a rtable 1 to the pass directive for that specific internal host like so

pass quick inet proto tcp from <FOO> to any port $allowed_tcp_ports $tcp_state rtable 1
pass quick inet proto udp from <FOO> to any port $allowed_udp_ports $udp_state rtable 1

no other changes for PF are needed, i.e. other pass directives do not need rtable 0...

2. adding additional routing tables is very easy and does not require kernel recompile, contrary to some online resources (maybe was true for earlier versions, I am on 12.1 RELEASE.
Just add

net.fibs="some integer"

to /boot/loader.conf and reboot

3. then - I needed to modify the opvn file for the 2nd connection on which client FOO needs to see the internets (tun1 in my case).

route-noexec
pull
pull-filter accept "route-gateway"
script-security 2
route-up /usr/local/etc/openvpn/route_up_foo.sh

4. the /usr/local/etc/openvpn/route_up_foo.sh does the hack :

setfib 1 route add 0.0.0.0/1 $route_vpn_gateway
setfib 1 route add 128.0.0.0/1 $route_vpn_gateway

the 2nd VPN connection also establishes some new routes in rtable 0, but for traffic from FOO it is important that once pf pushes it towards rtable 1, it is then routed to all possible destinations via tun1 gateway.

I should say that this solution was inspired by this