PF pf and fib routing for wireguard

I use the net/wireguard port for my VPN needs but have a problem if I want to connect to the same IP as the endpoint through the tunnel, because a static route is automatically added for obvious reasons to send this traffic directly rather than through the VPN.

Now one way around this is to use multiple fibs. So I can use the config Table = 1 in wireguard which makes it use an alternative routing table. This means I can do things like setfib 1 telnet 10.1.2.3 80 and it will route via the VPN, or remove the setfib command and it will route via the internet.

My question is, can anyone tell me how to use pf rules to make this happen automatically without the use of the setfib command. I've tried this at the bottom of my ruleset but that just results in broken connectivity:

Code:
pass out log rtable 1
pass out log proto udp to any port 51820 rtable 0

The intention of this is that I thought it would route all traffic through fib 1 except for udp port 51820 which would use fib 0. Doesn't appear to work though.

Has anyone done similar to this? How have you made it work? On Linux they use fwmark for things like this. I guess I'm on the right track but am doing something wrong.
 
I am trying to do something similar to make wireguard routing options more flexible on FreeBSD.

Some items of interest:
1) FWMark in wireguard is mapped to SO_USER_COOKIE socket option in setsockopt(2) on FreeBSD
2) SO_USER_COOKIE is supported in ipfw(8) syntax using "sockarg" token, and wireguard traffic marked with fwmark will match a sockarg rule. I have not seen evidence of support for this in pf firewall yet, but I have not looked exhaustively. ipfw sockarg processsing is disabled on ipv6 packets in-kernel at the moment, but works for ipv4. I will be hopefully posting a patch in bugzilla when I am done testing.
3) From my testing with the 20200121 version of wireguard, the Table option in the wg-quick configuration file alone does not provide a complete solution since the tun(4) interface is not created with the fib defined, and not all of the "route add" commands in wg-quick use the fib syntax from Table. I am experimenting to find a solution that at least works on 12-stable. If I find something that is useful, I will post a PR upstream to wireguard. The FreeBSD package/port maintainer seems to be keeping up with the wireguard upstream releases quickly.
4) Post your config files for wireguard (removing keys as needed). You should be able to use the inside tunnel ip address to force a connection through the vpn instead of the outside of tunnel ip address. This should be the "Address=" line under [Interface] on the remote side. You should not really need pf to do this unless you are trying to do something unusual.

As to proper syntax to use fibs in pf with wireguard, sorry I have not dug into that scenario yet.

Good Luck, and if anyone else has ideas, please share.
 
Thanks. That's some interesting information there. I could potentially move to ipfw if that can solve the problem. I'll have to have a read of the man page.

Unfortunately it's IPv6 that is causing me the problem. IPv4 isn't an issue. The reason being my server uses a private rfc1918 IP address which is NAT'd to a public IP, so because these don't match wireguard can happily route to the private IP through the tunnel.

With IPv6 I have a global IP on the server and there is no NAT involved. This means wireguard adds a static host route for that IP to avoid it being tunneled. And then I can't send traffic to it through the tunnel.

With openvpn I got around this by having two IPv6 IPs, one used for the VPN endpoint, and one used for the other services on the server like ssh etc. But this doesn't work with wireguard as the wireguard-go implementation on FreeBSD doesn't support "sticky-sockets". So the inbound traffic comes into the second IP, but wireguard responds using the first IP.

My config is here. The problem is because example.com resolves to 2001:db8:3c4d:15::10 this IP gets added as a /128 host route to the real interface to prevent routing loops through wg0. And then when you try to ssh example.com it fails. My current workaround is to use 4.example.com as the endpoint which only resolves to the IPv4 public IP. But I would like to also get IPv6 to the endpoint working if possible.

Code:
[Interface]
PrivateKey = ...
Address = 10.0.1.2/24, 2001:db8:3c4d:15::/64
DNS = 10.0.0.10, 2001:db8:3c4d:15::10

[Peer]
PublicKey = ...
PresharedKey = ...
Endpoint = example.com:51820
AllowedIPs = 0.0.0.0/0, ::/0
 
You can still use two ipv6 addresses for the server side of wireguard (like you do with OpenVPN). Sticky sockets should not be needed. Address 1 is assigned to physical interface (say em0), and Address 2 is assigned to the Wireguard tun()interface (say wg-server0). The key here is not to have both ipv6 addresses assigned to physical interfaces, and control your subnet mask/CIDR. On the server side config, use AllowedIPs=2001:db8::2/128 for the [Peer] (use /128 so you do not have a conflict with the other ipv6 address if they are both in the same /64 (likely).

Inside the tunnel address family does not have to match outside the tunnel address family with wireguard. It will happily tunnel ipv6 in ipv4 or ipv4 in ipv6, etc. In this example.com scenario, if this dns name has both A and AAAA records, the outside the tunnel comms to the other end will work just fine regardless of address family as long as there is end-to-end connectivity for UDP port 51820 to the server for both address families. Just ensure that you have UDP port 51820 open in your firewall for both your ipv4 remote endpoint , and your ipv6 remote endpoint in addition to NAT port forwarding rules if required by your network setup. Wireguard always does a ipv4/ipv6 wildcard bind to all interfaces (technically one socket per address family) for the ListenPort= configured. It will happily switch back and forth between interfaces and addresses depending on what address family is used to connect inbound outside the tunnel interface. In a normal internet facing server, this "bind everywhere" would be a concern, but since wireguard has a strong crypto-based security model, the risk is minimal, and you can always add firewall rules to add layers of protection as desired. As a side note, the Android and Windows builds of wireguard do not re-resolve dns names as aggressively as I would like, and dual-stack dns names for Endpoint configs frequently do not result in expected or desired behaviors. In these cases, it is always easier to use an ip address for initial troubleshooting to avoid looking at multiple problems simultaneously.

In addition, it is easier to start troubleshooting routing and comms issues if you start out with non-default routes (routes only for inside-tunnel ip ranges. The moment you use ::/0 or 0.0.0.0/0 in your AllowedIPs block, this can kill existing ssh connections, and disrupt other things since wg-quick/wg does both routes and allow ACLS based upon this single configuration line. There are tricks that allow you to impact the ACL list without impacting the routes by using PostUp configuration lines. The included wg-quick(8) bash script swallows most error conditions when adding/deleting routing entries by using the -q (quiet) command line (I hate this for troubleshooting reasons). This is some of the logic I am trying to find more elegant solutions around and create a PR for upstream. I also like to add a PostUp command to ping the other side of the tunnel so I can A) see traffic immediately (or not), and B) wireguard has completed the handshake right away so that is visible to wg(8) show command under "latest handshake" / "transfer" status.

Note: ListenPort= and Endpoint= configuration lines are for outside the tunnel, Address= and AllowedIPs= configuration lines are for inside the tunnel. AllowedIPs does two things: Routes and ACLs.

/etc/wireguard/Postup-client-example.conf
[Interface]
Privatekey = ABCD...
Address = 2001:db8::2/64, 10.0.0.2/24
# Allow any address coming from the other side of the tunnel (ipv4+ipv6)
PostUp = wg set Postup-client-example peer EFGH allow-ips ::/0,0.0.0.0/0
PostUp = ping6 -c 2 2001:db8::1

[Peer]
PublicKey = EFGH...
Endpoint = dualstack.example.com:51820
# Use /128 (ipv6) or /32 (ipv4) if all you need is point-to-point tunnel.
AllowedIPs = 2001:db8::/64, 10.0.0.0/24

Confirm your listening ports on the server:
#sockstat -l | grep -i wireguard
root wireguard- 83188 8 stream /var/run/wireguard/server0.sock
root wireguard- 83188 11 udp4 *:51820 *:*
root wireguard- 83188 12 udp6 *:51820 *:*


/etc/wireguard/wg-server0.conf (configuration file on server peer side)
# Remote endpoint with listen socket on pre-defined UDP port (aka server)
[Interface]
PrivateKey = WERT...
Address = 10.0.0.1/24, 2001:db8::1/64
ListenPort = 51820

[Peer]
PublicKey = YUIO...
AllowedIPs = 2001:db8::2/128, 10.0.0.2/32

/etc/wireguard/wg-client0.conf (configuration file on client peer side)
# Local endpoint with listen socket on dynamic UDP port
[Interface]
PrivateKey = HJKL...
Address = 2001:db8::2/64, 10.0.0.2/24

[Peer]
PublicKey = VBNM...
Endpoint = dualstack.example.com:51820
AllowedIPs = 2001:db8::/64, 10.0.0.0/24

Here is the quick and dirty patch to support ipv6 for sockarg in kernel ipfw:

I think this feature for ipfw sockarg on ipv6 was just disabled due to lack of testing pre-commit, but it seems to work in my limited experience thus far. Obviously, requires compiling a custom kernel to apply. YMMV.

Code:
diff --git a/sys/netpfil/ipfw/ip_fw2.c b/sys/netpfil/ipfw/ip_fw2.c
index 34ce208ee5f..2c31199bb0c 100644
--- a/sys/netpfil/ipfw/ip_fw2.c
+++ b/sys/netpfil/ipfw/ip_fw2.c
@@ -2557,8 +2557,6 @@ do {                                              \
                                struct inpcb *inp = args->inp;
                                struct inpcbinfo *pi;

-                               if (is_ipv6) /* XXX can we remove this ? */
-                                       break;

                                if (proto == IPPROTO_TCP)
                                        pi = &V_tcbinfo;

Apologies in advance for the rambling stream of consciousness, and any typos.
 
Thanks for the rambling stream of consciousness! I'm in the process of switching the firewall on my laptop and server back to using ipfw rather than pf. I think I'm mostly there, just a few loose ends to tie up. Then I'm going to test out this sockarg SO_USER_COOKIE support to see if that does anything for me. If I can make the laptop send traffic tagged with that sockarg via lagg0 rather than wg0 then this could work and looks interesting.

Don't suppose there are any snippets of config in ipfw for doing this? I see that the value of the option is put into a tablearg value. I guess you then use a table to direct the routing somehow. Will have to do some more reading of the man page.

The problem is the code you pasted there. My use case is specifically with IPv6 as the endpoint so I need that working. Grrrr. I could recompile the kernel but it's not ideal doing that.
 
Back
Top