Unable to get FreeBSD to NAT as a Gateway Router

[pacman2011]

I cannot get FreeBSD to correctly NAT / forward packets while acting as my front-facing firewall and gateway. Hopefully someone here can assist me in fine tuning this setup.

Problem: FreeBSD forwards packets from interface_0 to interface_1 correctly - but does not change the source IP of the packet. This causes the packets sent to the internet to have internal (non-routable) IP addresses.

Background: I am setting up my new network at my house for personal use. Being the nerdy guy I am, I want to implement my own manageable (Firewall / VPN / DNS / IDS) server on the outside of my internal network. I am currently running FreeBSD 9.1-RELEASE on a box that is going to act as the gateway for my private internal network to the external internet. I have attached a quick network diagram to assist describing this.

Question 0: Why won't FreeBSD NAT packets correctly for the wireless router? The FreeBSD gateway can access the internet, and anything 'behind' it. Anything 'behind' the FreeBSD gateway can access up to the FreeBSD gateway, on either IP.
I have included below:

1. Network Diagram
2. /etc/rc.conf
3. /etc/pf.conf
4. tcpdump that shows the external interface.

What am I missing?

# /etc/rc.conf
# NETWORKING - IPv4 configuration
ifconfig_bge0="inet 192.168.2.1 netmask 255.255.255.0"
ifconfig_bge1="inet A.B.C.D netmask 255.255.224.0"
defaultrouter="E.F.G.H"
static_routes="internal"
route_internal="-net 192.168.1.0/24 192.168.2.2"
gateway_enable="YES"

# Firewall Configuration
pf_enable="yes"                    # Enables Packet Filter (PF) Firewall
pf_rules="/etc/pf.conf"            # Sets the location of the pf rules file
pflog_enable="yes"                 # Enables Packet Logging from pf
pflog_logfile="/var/log/pf/pf.log" # Destination for the log file

# /etc/pf.conf
# Variables
internal_if = "bge0"
internal_gw = "192.168.2.2"
external_if = "bge1"

# Tables
table <internal> const { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 }

# Options
set skip on lo
set block-policy drop
set fingerprints "/etc/pf.os"
set optimization conservative

# Translation
no rdr on { lo0, lo1 } from any to any
nat on $external_if from <internal> to any -> ($external_if)

# Packet Filtering - Testing purpose only 
pass all

# tcpdump -ni bge1 icmp - Capture all ICMP packets on the EXTERNAL Interface.
# From behind Wireless Router
IP 192.168.2.2 > E.F.G.H: ICMP echo request, id 30161, seq 0, length 64
IP 192.168.2.2 > E.F.G.H: ICMP echo request, id 30161, seq 1, length 64
IP 192.168.2.2 > E.F.G.H: ICMP echo request, id 30161, seq 2, length 64

# From FreeBSD Gateway
IP A.B.C.D > E.F.G.H: ICMP echo request, id 30161, seq 0, length 64
IP E.F.G.H > A.B.C.D: ICMP echo reply, id 30161, seq 0, length 64

IP A.B.C.D > E.F.G.H: ICMP echo request, id 30161, seq 1, length 64
IP E.F.G.H > A.B.C.D: ICMP echo reply, id 30161, seq 1, length 64

IP A.B.C.D > E.F.G.H: ICMP echo request, id 30161, seq 2, length 64
IP E.F.G.H > A.B.C.D: ICMP echo reply, id 30161, seq 2, length 64

Question 1: Kind of a side question, MUCH more interested in the above.
Do I need to keep the FreeBSD gateway and the wireless router on different sub-nets? Could I make the FreeBSD gateway 192.168.1.2 and the wireless router 192.168.1.1?
NOTE: For reasons I won't get into, the wireless router MUST remain as 192.168.1.1

 

[J65nko]

ifconfig_bge0="inet 192.168.2.1 netmask 255.255.255.0"

So the network is 192.168.2.0/24. But your table containing the networks that need to be NATted is different:

192.168.0.0/16

If you change your NAT rule

nat on $external_if from <internal> to any -> ($external_if)

into

nat on $external_if from ! ($external_if) to any -> ($external_if)

you cover all cases and can get rid of that "<internal>" table.

 

[kpa]

I would turn the wireless router into a simple access point that just acts as a bridge between the wired and wireless networks. This can be done even if the firmware of the wireless router does not directly support bridging. You basically set up the LAN address of the wireless router to the same network with the LAN address of your FreeBSD router and turn DHCP off on it. You then assign the WAN address statically for the wireless router from a network that is never going to conflict with the other networks used. After this the WAN port on wireless router remains unused and you plug all wired equipment including the "upstream" cable to the FreeBSD router to the LAN switch of the wireless router.

 

[pacman2011]

Thank you to everyone who posted! Seriously, you guys make me proud to run FreeBSD.

That said - Progress! Although I don't know why x( I have a sneaking suspicion that PF was not enabled. :\ Regardless, after a reboot I am now up and running.

What made me notice that PF was not actually up and running was that the gateway stopped forwarding packets. Running the following directly after a reboot:

[root@gatekeeper]~ # sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 0
[root@gatekeeper]~ # sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1
[root@gatekeeper]~ # sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1

shows that the gateway_enable line in rc.conf (reproduced below) is not taking effect on boot, for whatever reason.

Questions:

1. Where should I look to find out why my

   gateway_enable="YES"

   is not taking effect on boot?
2. I'm not entirely sure that my PF is running on boot. I have to hit /etc/rc.d/pf start before my NAT will work - even though I get all the expected output from pfctl and from /etc/rc.d/pf status. How can I check that PF is in[]fact running on boot with the correct ruleset? If the above is the correct way to do it, why does my NAT fail until I kick PF in the pants?

/etc/rc.conf
Note: the IP address, default router, and netmask for em0 have been redacted - They are correct and function as expected.

hostname="gatekeeper"

#============#
# Networking #
#============#
ifconfig_em1="inet 192.168.2.1 netmask 255.255.255.0"   # LAN IP-Address
ifconfig_em0="inet A.B.C.D netmask X.X.X.X"             # WAN IP-Address
defaultrouter="E.F.G.H"                                 # Default Router
gateway_enable="YES"                                    # Enable IP-Forwarding

#========================#
# Firewall Configuration #
#========================#
pf_enable="YES"                     # Enable Packet Filter (PF) Firewall
pf_rules="/etc/pf.conf"             # Set the location of the pf rules file
pflog_enable="YES"                  # Enables Packet Logging from pf
pflog_logfile="/var/log/pf/pf.log"  # Destination for the log file

#=================#
# ENABLED DAEMONS #
#=================#
ftpproxy_enable="YES"       # Enable FTP-Proxy (Loophole for Access through PF)

unbound_enable="YES"        # Enable Unbound (DNS)

syslogd_enable="YES"        # Enable SysLog (System Logging)
syslogd_flags="-4 -ss -vv"  # -4  --> Forces the use of IPv4 addresses only
                            # -ss --> No socket for client/server setup
                            # -vv --> Prints the facility.level for each message

openntpd_enable="YES"       # Enables OpenNTPD (Time-Keeping)
openntpd_flags="-s"         # -s  --> Set time on boot (wait 15 sec for server)

sshd_enable="YES"           # Enables SSH (Secure Shell)
sshd_flags="-4"             # -4  --> Forces the use of IPv4 addresses only

#==================#
# DISABLED DAEMONS #
#==================#
portmap_enable="NO"   # Do not run the portmap daemon
sendmail_enable="NO"  # Do Not run the sendmail daemon

#=============#
# File System #
#=============#
dumpdev="AUTO"       # Enable crash dumps
dumpdir="/var/crash" # Directory where crash dumps are to be stored
zfs_enable="YES"     # Enable the ZFS_File Manager.

/etc/pf.conf

#===========#
# Variables #
#===========#
int_if = "em1"
ext_if = "em0"

#=========#
# Options #
#=========#
set skip on lo0
set block-policy drop

#=============#
# Translation #
#=============#
nat on $ext_if inet from $int_if:network:0 to any -> ($ext_if)

#===========#
# Filtering #
#===========#
block in log on $ext_if all

pass on $ext_if from ($int_if:network) to any keep state

 

[pacman2011]

So the network is 192.168.2.0/24. But your table containing the networks that need to be NATted is different:

192.168.0.0/16

Thank you for the reply! This was not the cause of the problem. The rule using the internal table will match on the network 192.168.2.0/24 because that network is within the larger subnet 192.168.0.0/16, which is in the table. That said - your suggestion still made sense to me - so I dropped that <internal> table and went with the following:

nat on $ext_if inet from $int_if:network to any -> ($ext_if)

Which worked once I set my net.inet.ip.forwarding variable to 1 and actually started PF.

Now if I only knew why it will not remain that way when I reboot, I'll be all set.

 

[pacman2011]

You basically set up the LAN address of the wireless router to the same network with the LAN address of your FreeBSD router and turn DHCP off on it. You then assign the WAN address statically for the wireless router from a network that is never going to conflict with the other networks used.

Thanks - but I believe that is kind of what I have done, and why the 192.168.2.0/24 network exists on my setup.

            Gateway           Free_BSD Box                          Router
Internet----E.F.G.H----A.B.C.D============192.168.2.1----192.168.2.2======192.168.1.1----Network

 

[kpa]

Well no, you should have the same network 192.168.2.0/24 on both sides of the wireless router that is actually made to operate as a simple wireless access point.

 

[pacman2011]

Ah - I see what you mean. I'll look into setting that up once I get the rebooting issue figured out.
Thank you for the feedback.

 

[J65nko]

RE: failure of starting pf

I noticed that when I use dhcp to assign an IP address, that sometimes pf does not start. The solution is then to use

ifconfig_re0="SYNCDHCP"

instead of

ifconfig_re0="DHCP"

This prevents the start up script from exiting without getting an IP address. pf needs interfaces that have an IP address.

Many years ago on bsdforums.org I helped somebody who had a strange problem, where he had to use dhcp, but that failed because of a duplex full-duplex mismatch of the NIC and a switch.
By setting the NIC to duplex in an /etc/start_if.xl0 we managed to solve that issue.

You could create a /etc/start_if.bge1 with something like this:

(date ; ifconfig ) >>/var/log/ifconfig.log
echo Sleeping for 10 seconds ....
sleep 10
(date ; ifconfig ) >>/var/log/ifconfig.log
sleep 1
(date ; netstat -rn -f inet) >>/var/log/ifconfig.log

Maybe the delay of 10 seconds helps, if not the contents of /var/log/ifconfig.log could give us a clue.

 

[Chris_H]

Which worked once I set my net.inet.ip.forwarding variable to 1 and actually started PF.

Now if I only knew why it will not remain that way when I reboot, I'll be all set.

You need to set that in sysctl.conf(5) eg;

/etc/sysctl.conf

net.inet.ip.forwarding=1   # previously 0

Then it will remain permanent through reboots. Remove/comment the line, should you ever want it to revert to default.

HTH

--Chris

 

[SirDice]

Remove net.inet.ip.forwarding from sysctl.conf, it's already being set with gateway_enable in rc.conf. Having both with cause confusion (besides the fact that everybody expects it to be set in rc.conf).

 

[Chris_H]

SirDice
Odd. I just setup a wifi gateway/router, and found that only setting gateway_enable="YES" in rc.conf(5) didn't always guarantee net.inet.ip.forwarding=1 stuck. So I added it. Mind you; I'm on -CURRENT (12), and ifconfig(8), and the network drivers have recently been going through some heavy changes. But I felt, given my experiences, and the fact that the OP indicated similar findings. That my reply was appropriate. No harm done adding it.

--Chris