IPFW IPFW kernel NAT port redirection not working

Hi everyone. First time poster, long time lurker. I am out of ideas on how to get port direction working with IPFW. Other than port redirection, the system works well and has been my workhorse for several years. My end goal is to expose an NGINX reverse proxy I have running in a jail on 192.168.0.2. When I try to run curl example.com I get:

Code:
% curl example.com
curl: (7) Failed to connect to example.com port 80: Connection refused

I thought this page was a good example of a working system so I tried to tailor my firewall rules like this, but it didn't seem to help me:
Thread ipfw-nat-stateful-redirect-of-a-port.58753

Here is a basic description of my system:
  • FreeBSD 12.1, generic kernel
  • single WAN port, system acting as a gateway to remaining interfaces
  • IPFW firewall using the kernel based NAT
  • Firewall by default blocks all incoming and outgoing traffic.
  • NGINX listening on port 80. Haven't enabled SSL or anything yet.

Things I've tried:
  • My domain name A records are working properly. I can SSH into the machine itself from the outside using the domain name.
  • I tried changing the port on NGINX to listen on port 8080 to make sure my ISP wasn't blocking the standard 80 port, but this resulted in the same Connection refused message
  • I can SSH into the jail from the LAN and I can pull receive data from the wider internet, so I think the networking here is OK. Running curl on the jail from the LAN I get traffic out so NGINX is running properly.
  • To rule out the jail altogether, I tried redirecting an arbitrary high number port to SSH into another machine on the LAN. Got the same Connection refused type message back here.
  • I tried using the userland natd with IPFW, but I get the same results. I made the necessary modifications for getting natd running in the rc.conf file and made a natd.conf with the port redirection commands.
  • I ran tcpdump -i igb0 tcp port 80 -vv and tried curl example.com and I saw no traffic at all. What is interesting though is if I SSH into this machine from the outside, when I run tcpdump on that port, I also see no traffic on the WAN interface, so I'm probably not doing this right

Below are the relevant configuration files:

rc.conf:
Code:
ifconfig_igb0="DHCP -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"                                                                                                                                                                                                                                                       
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable                                                                                                                                                                                                                 
dumpdev="AUTO"                                                                                                                                                                                                                                                         
gateway_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb1 addm igb2 addm igb3 addm igb4 addm igb5 addm igb6 addm igb7"
ifconfig_igb1="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb2="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb3="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb4="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb5="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb6="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb7="up -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_bridge0_alias0="inet 192.168.0.1 netmask 255.255.255.0"
dhcpd_enable="YES"
dhcpd_ifaces="bridge0"
unbound_enable="YES"
firewall_enable="YES"
firewall_logging="YES"
firewall_script="/etc/ipfw.rules"
# FIXME: testing out in kernel NAT                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
firewall_nat_enable="YES"
# FIXME: put NATD back if in kernel NAT doesn't work out                                                                                                                                                                                                                     
natd_enable="NO"
#natd_interface="igb0"                                                                                                                                                                                                                                                         
#natd_flags="-dynamic -m -f /etc/natd.conf"                                                                                                                                                                                                                                   
mountd_enable="YES"
nfs_server_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
rpcbind_enable="YES"
mountd_flags="-r"
inetd_enable="YES"
ifconfig_igb0="DHCP"
jail_enable="YES"

/etc/ipfw.rules
Bash:
#!/bin/sh                                                                                                                                                                                                                                                                     

# Set rules command prefix                                                                                                                                                                                                                                                     
cmd="/sbin/ipfw"
pif="igb0"     # interface name of NIC attached to Internet                                                                                                                                                                                                                   
skip="skipto 400"
isp_dhcp="xxx.xxx.xxx.xxx"
dns="1.1.1.1, 9.9.9.9"
outgoing_tcp_ports="1194, 8883, 80, 8080, 443, 22, 50683, 8245, 8333"
outgoing_udp_ports="1194, 500, 51820, 5090, 6000-29999, 80, 8080, 443, 123"
server_incoming_tcp_ports="50683"
client_incoming_tcp_ports="443, 80"
client_incoming_tcp_ports="443, 80"
# Flush out the list before we begin.                                                                                                                                                                                                                                         
$cmd -f flush

# configure NAT                                                                                                                                                                                                                                                               
$cmd nat 1 config if $pif unreg_only reset \
redirect_port tcp 192.168.0.2:80 80 \
redirect_port tcp 192.168.0.2:443 443 \

# No restrictions between LAN clients. Heaviest traffic first (the bridge)                                                                                                                                                                                                     
$cmd add 10 allow all from any to any via bridge0
$cmd add 11 allow all from any to any via lo0
$cmd add 12 allow all from any to any via igb1
$cmd add 13 allow all from any to any via igb2
$cmd add 14 allow all from any to any via igb3
$cmd add 15 allow all from any to any via igb4
$cmd add 16 allow all from any to any via igb5
$cmd add 17 allow all from any to any via igb6
$cmd add 18 allow all from any to any via igb7
$cmd add 19 allow all from any to any via tap0
$cmd add 20 allow all from any to any via epair0a

# catch spoofing from outside                                                                                                                                                                                                                                                 
$cmd add 50 deny ip from any to any not antispoof in recv $pif

# NAT diversion                                                                                                                                                                                                                                                               
$cmd add 59 nat 1 ip4 from any to any in recv $pif  # NAT any inbound packets                                                                                                                                                                                                 
$cmd add 60 reass all from any to any in # reassemble inbound packets                                                                                                                                                                                                         

# check all other conditions with the dynamic rules table                                                                                                                                                                                                                     
$cmd add 61 check-state

###################################################################                                                                                                                                                                                                           
# outbound connection rules                                                                                                                                                                                                                                                   
###################################################################                                                                                                                                                                                                           

# Allow access to public DNS over TLS                                                                                                                                                                                                                                         
$cmd add 100 $skip tcp from me to $dns 853 out xmit $pif setup keep-state

# Allow access to ISP's DHCP server for cable/DSL configurations.                                                                                                                                                                                                             
$cmd add 101 $skip udp from me to $isp_dhcp 67 out xmit $pif keep-state

# Allow good outgoing TCP, UDP, and ICMP                                                                                                                                                                                                                                       
$cmd add 110 $skip tcp from any to any $outgoing_tcp_ports out xmit $pif setup keep-state
$cmd add 111 $skip udp from any to any $outgoing_udp_ports out xmit $pif keep-state
$cmd add 112 $skip icmp from any to any out xmit $pif keep-state

# deny and log all other outbound connections                                                                                                                                                                                                                                 
$cmd add 199 deny log all from any to any out xmit $pif

/etc/systcl.conf
Code:
net.inet.tcp.tso="0"
net.inet.ip.fw.one_pass="0"

I'm at a loss at this point.
 
Hello zettawatt,

here is my /etc/ipfw.conf and port redirect is working for me

Code:
$ more /etc/ipfw.conf
#!/bin/sh

ipfw -q -f flush        # Flush out the list

# Rules command prefix
ks="keep-state"
skip="skipto 1000"
allowed="ssh,http,pop3,ntp,https,smtps,imaps"
amr="ipfw -q add"
loc="lo0"               # Loopback interface
wan="tun0"              # WAN interface
lan="bge0"              # LAN interface
dns="208.67.222.123,208.67.220.123"

ipfw disable one_pass
ipfw -q nat 1 config if $wan same_ports unreg_only reset \
   redirect_port tcp 192.168.100.10:http http \
   redirect_port tcp 192.168.100.10:https https

# Allow all traffic on LAN and on the loopback interface
$amr 0100 allow all from any to any via $lan
$amr 0150 allow all from any to any via $loc

# The reassemble rule
$amr 0200 reass all from any to any in

# In-kernal NAT
$amr 0250 nat 1 ip from any to any in via $wan

# Allow the packet through if it matches an existing entry in the dynamic rules table
$amr 0300 check-state

# Allow access to OpenDNS only
$amr 0350 $skip all from any to $dns domain out via $wan $ks

# Allowed ports
$amr 0400 $skip all from any to any $allowed out via $wan $ks

# Port Redirection
$amr 0450 $skip all from any to 192.168.100.10 http,https in via $wan $ks

# Kids PC
$amr 0451 $skip all from 192.168.100.1 to any out via $wan $ks
$amr 0452 $skip all from 192.168.100.2 to any out via $wan $ks
$amr 0453 $skip all from 192.168.100.3 to any out via $wan $ks

# Allow ping
$amr 0500 $skip icmp from any to any out via $wan $ks

# Deny and log everything else
$amr 0550 deny log all from any to any

# Skipto location for outbound stateful rules
$amr 1000 nat 1 ip from any to any out via $wan

$amr 1050 allow ip from any to any
 
Your firewall rule set seems to be incomplete - did you copy and paste the whole set?

It is missing the outgoing nat rule which should have number 400, since this is where the skipto directive would pass matching traffic to. It is also missing a rule for allowing external traffic coming in on port 80 and 443, something alike:

Code:
# Rules for allowing dial-in calls to services which are listening on a LAN interface behind the NAT
/sbin/ipfw -q add 160 $skip tcp from any to any 80,443 in recv $pif setup keep-state
 
Sorry guys, I didn't copy the whole rules file for some reason, ugh. Let me try that again:

Code:
#!/bin/sh

# Set rules command prefix
cmd="/sbin/ipfw"
pif="igb0"     # interface name of NIC attached to Internet
skip="skipto 400"
isp_dhcp="xxx.xxx.xxx.xxx"
dns="1.1.1.1, 9.9.9.9"
outgoing_tcp_ports="1194, 8883, 80, 8080, 443, 22, 50683, 8245, 8333"
outgoing_udp_ports="1194, 500, 51820, 5090, 6000-29999, 80, 8080, 443, 123"
server_incoming_tcp_ports="50683"
client_incoming_tcp_ports="443, 80"
client_incoming_udp_ports="443, 80"
# Flush out the list before we begin.
$cmd -f flush

# configure NAT
$cmd nat 1 config if $pif unreg_only reset \
redirect_port tcp 192.168.0.2:80 80 \
redirect_port tcp 192.168.0.2:443 443 \

# No restrictions between LAN clients. Heaviest traffic first (the bridge)
$cmd add 10 allow all from any to any via bridge0
$cmd add 11 allow all from any to any via lo0
$cmd add 12 allow all from any to any via igb1
$cmd add 13 allow all from any to any via igb2
$cmd add 14 allow all from any to any via igb3
$cmd add 15 allow all from any to any via igb4
$cmd add 16 allow all from any to any via igb5
$cmd add 17 allow all from any to any via igb6
$cmd add 18 allow all from any to any via igb7
$cmd add 19 allow all from any to any via tap0
$cmd add 20 allow all from any to any via epair0a

# catch spoofing from outside
$cmd add 50 deny ip from any to any not antispoof in recv $pif

# NAT diversion
$cmd add 59 nat 1 ip4 from any to any in recv $pif  # NAT any inbound packets
$cmd add 60 reass all from any to any in # reassemble inbound packets

# check all other conditions with the dynamic rules table
$cmd add 61 check-state

###################################################################
# outbound connection rules
###################################################################

# Allow access to public DNS over TLS
$cmd add 100 $skip tcp from me to $dns 853 out xmit $pif setup keep-state

# Allow access to ISP's DHCP server for cable/DSL configurations.
$cmd add 101 $skip udp from me to $isp_dhcp 67 out xmit $pif keep-state

# Allow good outgoing TCP, UDP, and ICMP
$cmd add 110 $skip tcp from any to any $outgoing_tcp_ports out xmit $pif setup keep-state
$cmd add 111 $skip udp from any to any $outgoing_udp_ports out xmit $pif keep-state
$cmd add 112 $skip icmp from any to any out xmit $pif keep-state

# deny and log all other outbound connections
$cmd add 199 deny log all from any to any out xmit $pif

###################################################################
# server inbound connection rules
###################################################################

# Allow traffic from ISP's DHCP server.
$cmd add 200 allow udp from $isp_dhcp to me 67 in recv $pif keep-state

# Allow server connections
$cmd add 201 allow tcp from any to me $server_incoming_tcp_ports in recv $pif setup limit src-addr 2

# Allow UDP fragments
$cmd add 202 allow udp from any to me in recv $pif frag

###################################################################
# LAN client inbound connections
###################################################################

$cmd add 300 $skip tcp from any to any $client_incoming_tcp_ports in recv $pif setup keep-state
$cmd add 301 $skip udp from any to any $client_incoming_udp_ports in recv $pif keep-state

###################################################################
# Final filtering and outbound NAT redirection
###################################################################

# Reject and log all other connections
$cmd add 399 deny log all from any to any via $pif

# skipto location for outbound stateful rules
$cmd add 400 nat 1 ip4 from any to any out xmit $pif

# allow anything else not already filtered out
$cmd add 401 allow ip from any to any
 
I kept tweaking this and all of a sudden it works! I need to step back through and see what I changed to get this running. I ended up having to restart this machine along the way for another reason. Maybe something else was conflicting and causing issues? I will post back when I root cause my issue for future reference.
 
Hi,

Probably I should start the new topic, buy maybe the problem is connected somehow?

I'm using natd for years. It works also on my server which I have upgraded last week to 12.1-RELEASE. Today I required to redirect port for test something and I've added to my /etc/natd.conf the line:
Code:
redirect_port tcp 192.168.64.252:80 3456
I've restarted daemon: service natd resrtart and there was no redirection :(

I've made some test, changed ipfw configuration but without success. Finaly I've made almost the simplest condiguration:
  • I've put to /etc/natd.conf
    Code:
    unregistered_only yes
    log_denied yes
    dynamic yes
    use_sockets yes
    same_ports yes
    redirect_port tcp 192.168.64.252:80 3456
  • then I've made:
    Code:
    ipfw -f flush
    ipfw add divert natd all from any to any via em0
    ipfw add pass all from any to any
  • and: service natd restart which run /sbin/natd -config /etc/natd.conf -n em0 according to my configuration in /etc/rc.conf:
    Code:
    natd_enable="YES"
    natd_interface="em0"
    natd_flags="-config /etc/natd.conf"
    gateway_enable="YES"
After all the redirection ... stil not working.
What have I missed?

Regards,
Marcin

PS. I have loaded ipfw.ko and ipdivert.ko. Do I need to load other modules, like libalias.ko?
 
Back
Top