Solved Can't get NAT working with pf on FreeBSD 13.1

dvl@

Developer
I'm trying to get NAT working in my basement. NAT is running fine on my gateway, but this is another situation. In this post, I'm trying to mix in enough detail without overloading.

EDIT: I did get the NAT working. In the end, I did not use it.

I have a 10.55.0.0/24 network in the basement. The gateway handles incoming traffic. One host inside the gateway has a VPN connection to a host at a colo facility. The goal of the VPN is to all that remote host to proxy incoming web requests to webservers in my basement. In short, I am relocating an existing proxy in my basement to a proxy in a colo.

Let's refer to these two hosts as the portal host (the one in the colo) and the basement host (the one... in my basement, seen below as r720-01).

Current status: both hosts are connected via a Wireguard tunnel. The portal host can ping any host in my basement network.

These are the PF rules in place on the basement host:

Code:
[r720-01 dan ~] % cat /etc/pf.conf
EXT_IF="ix0"
WG_IF="wg0"

LAN="10.55.0.0/24"

# e.g. 10.9.1.144
WG_LAN="10.9.1.0/24"

set skip on lo0
#set skip on $EXT_IF

scrub in all

#nat on $EXT_IF from $WG_LAN to any -> ($EXT_IF)

nat on $EXT_IF inet from $WG_LAN -> 10.55.0.59

pass in all
pass out all

That IP address I'm trying to NAT to is defined on the NIC:

Code:
[r720-01 dan ~] % ifconfig | grep 10.55.0.59
    inet 10.55.0.59 netmask 0xffffffff broadcast 10.55.0.59

Forwarding is enabled on this host:

Code:
[r720-01 dan ~] % sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1

And the wg0 NIC:

Code:
[r720-01 dan ~] % ifconfig wg0
wg0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.9.1.145 netmask 0xfffffffc broadcast 10.9.1.147
    groups: tun
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
    Opened by PID 88424
[r720-01 dan ~] %

This is the incoming ping request as seen on the basement host. ix0 is the main NIC on this host.

Code:
[r720-01 dan ~] % sudo tcpdump -ni ix0 host 10.9.1.144
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:31:38.540370 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 913, length 64
14:31:38.540677 IP 108.36.95.115 > 10.9.1.144: ICMP echo reply, id 56414, seq 913, length 64
14:31:39.603879 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 914, length 64
14:31:39.604183 IP 108.36.95.115 > 10.9.1.144: ICMP echo reply, id 56414, seq 914, length 64
^C
4 packets captured
54 packets received by filter
0 packets dropped by kernel
[r720-01 dan ~] %

This is what that ping looks like on the host being pinged:

Code:
14:32:23.112376 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 955, length 64
14:32:23.112424 IP 10.55.0.73 > 10.9.1.144: ICMP echo reply, id 17234, seq 955, length 64

Instead of 10.9.1.144 being seen on the network, I'd like to NAT that to 10.55.0.59.

Ideas please?
 
Last edited:
Code:
[r720-01 dan ~] % ifconfig | grep 10.55.0.59 
inet 10.55.0.59 netmask 0xffffffff broadcast 10.55.0.59
Shouldn't this have a /24 netmask instead of a /32? And is this attached to ix0?

I'm somewhat confused by this:
Code:
14:31:38.540370 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 913, length 64
14:31:38.540677 IP 108.36.95.115 > 10.9.1.144: ICMP echo reply, id 56414, seq 913, length 64
Why does the response come from an internet IP address? I expect to see a 10.55.0.73 source address here. Is the reply actually being returned through the VPN tunnel?
 
Shouldn't this have a /24 netmask instead of a /32? And is this attached to ix0?

It has a /32 because It is an alias, and yes, it is attached to ix0.

I'm somewhat confused by this:
Code:
14:31:38.540370 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 913, length 64
14:31:38.540677 IP 108.36.95.115 > 10.9.1.144: ICMP echo reply, id 56414, seq 913, length 64
Why does the response come from an internet IP address? I expect to see a 10.55.0.73 source address here. Is the reply actually being returned through the VPN tunnel?
Well, I didn't check my copy/paste closely did I. That's the public IP address of my gateway. That is interesting.

Correcting my post.

That is the gateway of my IP. There is a static route on that redirecting 10.9.1.0/24 to 10.55.0.59.

I could also add that to every host. Mind you, that routing problem goes away if I get the NAT working.
 
That is the gateway of my IP. There is a static route on that redirecting 10.9.1.0/24 to 10.55.0.59.

I could also add that to every host. Mind you, that routing problem goes away if I get the NAT working.

Route added to host being pinged:

Code:
[slocum dan ~] % sudo route add 10.9.1.0/24 10.55.0.59
add net 10.9.1.0: gateway 10.55.0.59 fib 0

The change in the tcpdump output:
Code:
16:05:24.322008 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 6241, length 64
16:05:24.322356 IP 108.36.95.115 > 10.9.1.144: ICMP echo reply, id 56414, seq 6241, length 64
16:05:25.385474 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 6242, length 64
16:05:25.385653 IP 10.55.0.73 > 10.9.1.144: ICMP echo reply, id 17234, seq 6242, length 64
16:05:26.386661 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 6243, length 64
16:05:26.386740 IP 10.55.0.73 > 10.9.1.144: ICMP echo reply, id 17234, seq 6243, length 64
16:05:27.449155 IP 10.9.1.144 > 10.55.0.73: ICMP echo request, id 17234, seq 6244, length 64
16:05:27.449225 IP 10.55.0.73 > 10.9.1.144: ICMP echo reply, id 17234, seq 6244, length 64

The lack of NAT remains.
 
Can you share both routing tables on the both ends of Wireguard hosts while the VPN is up.

The portal host in the colo:
Code:
[r720-02 dan ~] % netstat -nr4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            173.228.xxx.xxx    UGS        igb0
10.9.1.144         link#14            UHS         lo0
10.9.1.144/30      link#14            U           wg0
10.9.1.145         link#14            UHS         wg0
10.55.0.0/24       link#14            US          wg0
10.140.0.217       link#1             UH          lo0
127.0.0.1          link#7             UH          lo0
127.163.0.10       link#8             UH          lo1
127.163.0.25       link#10            UH          lo3
127.163.0.53       link#10            UH          lo3
127.163.0.80       link#9             UH          lo2
127.163.54.32      link#11            UH          lo4
173.228.xxx.xxx/29 link#1             U          igb0
173.228.xxx.xxx    link#1             UHS         lo0

The basement host:

Code:
[r720-01 dan ~] % netstat -nr4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.55.0.1          UGS         ix0
10.9.1.144         link#9             UHS         wg0
10.9.1.144/30      link#9             U           wg0
10.9.1.145         link#9             UHS         lo0
10.55.0.0/24       link#1             U           ix0
10.55.0.10         link#1             UH          lo0
10.55.0.18         link#1             UHS         lo0
10.55.0.23         link#1             UH          lo0
10.55.0.32         link#1             UH          lo0
10.55.0.33         link#1             UH          lo0
10.55.0.34         link#1             UH          lo0
10.55.0.35         link#1             UH          lo0
10.55.0.59         link#1             UH          lo0
127.0.0.1          link#5             UH          lo0
127.1.0.201        link#6             UH          lo1
 
That did it. Thank you.

Code:
[slocum dan ~] % sudo tcpdump -ni ix2 host 10.55.0.59
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:23:59.433436 IP 10.55.0.59 > 10.55.0.3: ICMP echo request, id 21088, seq 122, length 64
17:23:59.433490 IP 10.55.0.3 > 10.55.0.59: ICMP echo reply, id 21088, seq 122, length 64
17:24:00.481042 IP 10.55.0.59 > 10.55.0.3: ICMP echo request, id 21088, seq 123, length 64
17:24:00.481108 IP 10.55.0.3 > 10.55.0.59: ICMP echo reply, id 21088, seq 123, length 64
^C
4 packets captured
244 packets received by filter
0 packets dropped by kernel
[slocum dan ~] %
 
127.0.0.0/8 is not routable that's why. The clients (jails/vm) on colo site are using 127.163.0.0/16 subnet so the only option is to create another subnet for those clients for example 10.55.1.0/24 for that site and then create a static route on the default gateway on 10.55.0.1 -> 10.55.1.0/24 gw 10.55.0.59 so all clients from 10.55.0.0/24 will have access to 10.55.1.0/24 via gw 10.55.0.59 but he will need to redesign his network at colo site.
 
10.9.1.144/30 network is just a vpn tunnel with two endpoints and to be correct the two end points should have 10.9.1.145 and 10.9.1.146 ip addresses as 10.9.1.144 is the network address itself.
 
10.9.1.144/30 network is just a vpn tunnel with two endpoints and to be correct the two end points should have 10.9.1.145 and 10.9.1.146 ip addresses as 10.9.1.144 is the network address itself.
This is the basement box:

Code:
[r720-01 dan ~] % ifconfig wg0
wg0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.9.1.145 netmask 0xfffffffc broadcast 10.9.1.147
    groups: tun
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
    Opened by PID 88424

This is the colo portal box:

Code:
[r720-02 dan ~] % ifconfig wg0
wg0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.9.1.144 netmask 0xfffffffc broadcast 10.9.1.147
    groups: tun
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
    Opened by PID 683

Are you saying the Address in wg0.conf here should be 10.9.1.146?

Code:
[r720-02 dan ~] % sudo cat /usr/local/etc/wireguard/wg0.conf
[Interface]
Address = 10.9.1.144/30
...

If so, I think I now understand why when I revisit the subnet calculator I was using:
Code:
Usable Host IP Range:    10.9.1.145 - 10.9.1.146

If thats the case, I'll redo the tunnel.
 
For the tunnel endpoints it doesn't matter they could be /32 or even there's no need to be in the same network but it doesn't look good that's why it's better to try to be compliant. For example in Azure vpn they use apipa address which for me is a nonsense.
 
What would be compliant in this case please? I’ll be blogging about this and want to avoid spreading bad habits.
 
.145 and .146 for the tunnel endpoints.
After making the changes, the pings were coming from:

Code:
18:30:25.222766 IP 10.9.1.146 > 10.55.0.24: ICMP echo request, id 16291, seq 3275, length 64
18:30:25.222800 IP 10.55.0.24 > 10.9.1.146: ICMP echo reply, id 16291, seq 3275, length 64

I modified the pf nat rule from:

Code:
nat on $EXT_IF from 10.9.1.144/32 to 10.55.0.0/24 -> 10.55.0.59

to:

Code:
nat on $EXT_IF from 10.9.1.144/30 to 10.55.0.0/24 -> 10.55.0.59

Only the netmask changed in the from clause, from /32 to /30.

I restarted wireguard on both hosts, and restarted pf. The objective still works:

Code:
18:30:45.306599 IP 10.55.0.59 > 10.55.0.24: ICMP echo request, id 9230, seq 3294, length 64
18:30:45.306661 IP 10.55.0.24 > 10.55.0.59: ICMP echo reply, id 9230, seq 3294, length 64
 
@validgn
.145 and .146 for the tunnel endpoints.
Do you have other recommendations regarding this tunnel?

Can 10.55.0.59 be used so that everything coming out of that tunnel is from that address without using NAT?

I'm happy with the current solution, I'm just thinking of what you said: "10.9.1.144 is the network address itself."
 
As others already points out it's better to avoid using NAT if you can afford to create a different subnet for clients on your colo site. This will give you more flexibility to filter particular hosts on your colo site and on heavy loaded networks the NAT will be slow that's why routing is preferred.
 
The main client on the colo site will be an Nginx instance running as a proxy. It will be proxying to hosts in the basement.

I could have that listening on 10.9.2.0/24 (for example) - that subnet would then be the colo clients. Is that what you’re getting at?
 
Here's an example of what i'm thinking:
Untitled Diagram.drawio (1).png
 
Back
Top