I quite often find that for a number of reasons, it's much easier to use NAT for faux-internal networks (ie: a VPS with multiple jails but only 1 public IP), and for remote access VPNs. Especially remote access VPNs...
However I almost always end up getting things wrong, and I'm not sure why. The end result is that things that I want to be NAT'd are not.
My hypothetical example looks like this:
- A number of servers behind a router that we do not control (this is the external/internet-facing) network, let's call it 172.16.30.0/24, and on each server rc.conf sets the interface name of the external interface to "ext0".
- All servers have a second NIC which is the "internal" network, let's call that 10.10.10.0/24, and on each server rc.conf sets the interface name of the internal interface to "int0"
- Two servers have OpenVPN configured for remote access to the internal network (two for redundancy). Let's say each of those hand out client IPs in the range of 192.168.100.0/24 and 192.168.200.0/24 (ie: the "server" line in openvpn.conf is set like so: server 192.168.100.0 255.255.255.0 or server 192.168.200.0 255.255.255.0)
- There are additional parts of the "internal" network living on even more different subnets at a few VPS hosts and home/office networks where the VPN is handled not on individual clients, but on the router/firewall at those locations. I'm ignoring those for now, but keep in mind that keeping traffic flowing between these sites involves setting up static routes when the VPN client IP pools are not NAT'd, which without some kind of routing protocol is a total PITA and suboptimal.
OK, so part of that preface is to address "why NAT", beyond "because I want to and this is how I prefer to do it".
Of note:
- net.inet.ip.forwarding is set to "1"
- there is no "skip" directive on int0
So let's take a very basic example that is currently not working from one of the servers running OpenVPN for single remote hosts to access the internal network:
nat on int0 inet from 192.168.100.0/24 to any -> 10.10.10.2
int0 = internal interface of server
192.168.100.0/24 = OpenVPN client subnet
10.10.10.2 = main internal IP of the server
Variants that also do not work:
nat on tun0 inet from 192.168.100.0/24 to any -> 10.10.10.2
(tun0 = OpenVPN tunnel interface)
nat on int0 inet from 192.168.100.2/32 to any -> 10.10.10.2
and
nat on tun0 inet from 192.168.100.2/32 to any -> 10.10.10.2
(specifying only the IP of the remote OpenVPN client I'm troubleshooting)
How am I verifying this doesn't work? I do the following with each change:
- pfctl -f /etc/pf.conf to pick up the nat changes
- pfctl -sn to verify my changes are live
- a ping from the remote VPN client to another host on the internal network
- a tcpdump on the tun0 or int0 interfaces to see if the address has been translated or not
- a tcpdump on the internal host I'm pinging to see if the address has been translated or not
Here's the nat rule and example tcpdump output showing no translation when pinging from the VPN client to another host (10.10.10.3) on the internal LAN:
nat on int0 inet from 192.168.100.2 to any -> 10.10.10.2 (from pfctl -sn)
(from the host running OpenVPN server, listening on the tun0 interface)
[root@clweb3 /home/spork]# tcpdump -vvn -i tun0 src or dst 10.10.10.3
tcpdump: listening on tun0, link-type NULL (BSD loopback), capture size 262144 bytes
18:23:33.704884 IP (tos 0x0, ttl 64, id 40843, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 9, length 64
18:23:34.706749 IP (tos 0x0, ttl 64, id 36399, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 10, length 64
18:23:35.712629 IP (tos 0x0, ttl 64, id 21061, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 11, length 64
(from the host running OpenVPN server listening on the int0 interface)
[root@clweb3 /home/spork]# tcpdump -vvn -i int0 src or dst 10.10.10.3
tcpdump: listening on int0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:23:45.748698 IP (tos 0x0, ttl 63, id 18682, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 21, length 64
18:23:46.754720 IP (tos 0x0, ttl 63, id 5832, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 22, length 64
18:23:47.755645 IP (tos 0x0, ttl 63, id 7329, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 23, length 64
(from the host 10.10.10.3, running on the int0 interface)
[root@clweb4 /home/spork]# tcpdump -vvn -i int0 src or dst 10.10.10.3
tcpdump: listening on int0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:28:59.054396 IP (tos 0x0, ttl 63, id 43937, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 333, length 64
18:29:00.057061 IP (tos 0x0, ttl 63, id 63601, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 334, length 64
18:29:01.061593 IP (tos 0x0, ttl 63, id 31647, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 335, length 64
What I would be expecting if NAT were matching/working is that everywhere above where I see "192.168.100.2", I would instead see "10.10.10.2"
So that's where I'm stuck. I can see that the rule is not working to translate the packets, but as best as I can tell, it is setup correctly. What am I missing? I've been using pf since it didn't exist (ipf), so I'm thinking maybe something has changed since I spent large amounts of time working with pf years and years ago (this host is 13.3)?
My understanding of how this NAT rule breaks down:
nat on int0 inet from 192.168.100.2 to any -> 10.10.10.2
"nat on int0" = "int0 is the interface on which we apply this rule"
"from 192.168.100.2 to any" = "traffic heading out int0 with source 192.168.100.2"
"-> 10.10.10.2" = "rewrite the source address of matched traffic to 10.10.10.2"
Am I wrong? As noted I've tried this on "tun0" as well with no change.
However I almost always end up getting things wrong, and I'm not sure why. The end result is that things that I want to be NAT'd are not.
My hypothetical example looks like this:
- A number of servers behind a router that we do not control (this is the external/internet-facing) network, let's call it 172.16.30.0/24, and on each server rc.conf sets the interface name of the external interface to "ext0".
- All servers have a second NIC which is the "internal" network, let's call that 10.10.10.0/24, and on each server rc.conf sets the interface name of the internal interface to "int0"
- Two servers have OpenVPN configured for remote access to the internal network (two for redundancy). Let's say each of those hand out client IPs in the range of 192.168.100.0/24 and 192.168.200.0/24 (ie: the "server" line in openvpn.conf is set like so: server 192.168.100.0 255.255.255.0 or server 192.168.200.0 255.255.255.0)
- There are additional parts of the "internal" network living on even more different subnets at a few VPS hosts and home/office networks where the VPN is handled not on individual clients, but on the router/firewall at those locations. I'm ignoring those for now, but keep in mind that keeping traffic flowing between these sites involves setting up static routes when the VPN client IP pools are not NAT'd, which without some kind of routing protocol is a total PITA and suboptimal.
OK, so part of that preface is to address "why NAT", beyond "because I want to and this is how I prefer to do it".
Of note:
- net.inet.ip.forwarding is set to "1"
- there is no "skip" directive on int0
So let's take a very basic example that is currently not working from one of the servers running OpenVPN for single remote hosts to access the internal network:
nat on int0 inet from 192.168.100.0/24 to any -> 10.10.10.2
int0 = internal interface of server
192.168.100.0/24 = OpenVPN client subnet
10.10.10.2 = main internal IP of the server
Variants that also do not work:
nat on tun0 inet from 192.168.100.0/24 to any -> 10.10.10.2
(tun0 = OpenVPN tunnel interface)
nat on int0 inet from 192.168.100.2/32 to any -> 10.10.10.2
and
nat on tun0 inet from 192.168.100.2/32 to any -> 10.10.10.2
(specifying only the IP of the remote OpenVPN client I'm troubleshooting)
How am I verifying this doesn't work? I do the following with each change:
- pfctl -f /etc/pf.conf to pick up the nat changes
- pfctl -sn to verify my changes are live
- a ping from the remote VPN client to another host on the internal network
- a tcpdump on the tun0 or int0 interfaces to see if the address has been translated or not
- a tcpdump on the internal host I'm pinging to see if the address has been translated or not
Here's the nat rule and example tcpdump output showing no translation when pinging from the VPN client to another host (10.10.10.3) on the internal LAN:
nat on int0 inet from 192.168.100.2 to any -> 10.10.10.2 (from pfctl -sn)
(from the host running OpenVPN server, listening on the tun0 interface)
[root@clweb3 /home/spork]# tcpdump -vvn -i tun0 src or dst 10.10.10.3
tcpdump: listening on tun0, link-type NULL (BSD loopback), capture size 262144 bytes
18:23:33.704884 IP (tos 0x0, ttl 64, id 40843, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 9, length 64
18:23:34.706749 IP (tos 0x0, ttl 64, id 36399, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 10, length 64
18:23:35.712629 IP (tos 0x0, ttl 64, id 21061, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 11, length 64
(from the host running OpenVPN server listening on the int0 interface)
[root@clweb3 /home/spork]# tcpdump -vvn -i int0 src or dst 10.10.10.3
tcpdump: listening on int0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:23:45.748698 IP (tos 0x0, ttl 63, id 18682, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 21, length 64
18:23:46.754720 IP (tos 0x0, ttl 63, id 5832, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 22, length 64
18:23:47.755645 IP (tos 0x0, ttl 63, id 7329, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 23, length 64
(from the host 10.10.10.3, running on the int0 interface)
[root@clweb4 /home/spork]# tcpdump -vvn -i int0 src or dst 10.10.10.3
tcpdump: listening on int0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:28:59.054396 IP (tos 0x0, ttl 63, id 43937, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 333, length 64
18:29:00.057061 IP (tos 0x0, ttl 63, id 63601, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 334, length 64
18:29:01.061593 IP (tos 0x0, ttl 63, id 31647, offset 0, flags [none], proto ICMP (1), length 84)
192.168.100.2 > 10.10.10.3: ICMP echo request, id 45625, seq 335, length 64
What I would be expecting if NAT were matching/working is that everywhere above where I see "192.168.100.2", I would instead see "10.10.10.2"
So that's where I'm stuck. I can see that the rule is not working to translate the packets, but as best as I can tell, it is setup correctly. What am I missing? I've been using pf since it didn't exist (ipf), so I'm thinking maybe something has changed since I spent large amounts of time working with pf years and years ago (this host is 13.3)?
My understanding of how this NAT rule breaks down:
nat on int0 inet from 192.168.100.2 to any -> 10.10.10.2
"nat on int0" = "int0 is the interface on which we apply this rule"
"from 192.168.100.2 to any" = "traffic heading out int0 with source 192.168.100.2"
"-> 10.10.10.2" = "rewrite the source address of matched traffic to 10.10.10.2"
Am I wrong? As noted I've tried this on "tun0" as well with no change.
