IPFW IPFW in-kernel NAT does not work as expected

F

fixinit75

Hello. I am rather new to the firewall of FreeBSD and I am having issues with the in-kernel NAT.
I had configured everything according to the man pages and some internet resources.

My test infrastructure looks like this:

1718726211885.png

In the production environment the WAN interface will be directly connected to the internet and have a public IP, LAN will be replaced with tap0 interface of the OpenVPN.

I am expecting IPFW to allow local connections on ports 22 (ssh), 8480 (OpenVPN), all outbound traffic. The NAT is expecting to redirect the port 24 to port 22 (ssh) of my ubuntu VM. NAT should also allow all outbound traffic from LAN.

The actual behavior is the following:
- I can ping 8.8.8.8 from the ubuntu, but can't ping google.com. TCPdump shows that during the ping for google.com the dns requests are successfully translated by NAT and sent from vtnet0 with no answer from the internet
- I can ping both 8.8.8.8 and google.com from FreeBSD, but the kernel logs show that the the sendto permission is denied for NTP requests
- I can't connect to ubuntu via ssh from 192.168.10.0/24 (vtnet0). The packets do never get redirected to LAN. They get blocked by the rule 499
- I can't connect from ubuntu to any WAN devices with ssh, the packets are successfully translated by nat, they exit vtnet0 with no response from the internet

What am I doing wrong? I had looked for similar threads here and did some modifications to the firewall config, but had to revert everything as that didn't help me at all


The configs:
/boot/loader.conf
Code: 
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
#console="comconsole"
console="video"
ipfw_load="YES"
ipfw_nat_load="YES"
libalias_load="YES"
aesni_load="YES"
net.inet.ip.fw.default_to_accept="0"

/etc/rc.conf
Code: 
hostname="BSD-PFTEST"
ifconfig_vtnet0="DHCP"
defaultrouter="192.168.10.1"
ifconfig_vtnet1="192.168.127.1/24"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
qemu_guest_agent_enable=YES

ntpd_enable="NO"
ntpdate_enable="YES"
ntpdate_flags="-u"
fsck_y_enable="YES"
background_fsck="NO"

gateway_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
firewall_nat_enable="YES"
firewall_nat_interface="vtnet0"
firewall_logif="YES"

openvpn_enable="YES"
openvpn_if="tun tap"

/etc/sysctl.conf
Code: 
vfs.zfs.min_auto_ashift=12
net.inet.ip.fw.one_pass="0"
net.inet.tcp.tso="0"
Kernel config:
Code: 
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=50
options IPFIREWALL_NAT
options LIBALIAS
options ROUTETABLES="2"
options IPFIREWALL_NAT64
options IPFIREWALL_NPTV6
options IPFIREWALL_PMOD 
options HZ="1000"
/etc/ipfw.rules
Code: 
#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
ipfw="ipfw -q"
add="ipfw -q add"
pif="vtnet0"     # interface name of NIC attached to Internet
vpn="tap0" #interface name of OpenVPN
lan="vtnet1"
skip="skipto 1000"

$ipfw disable one_pass
$ipfw nat 1 config log if vtnet0 reset unreg_only same_ports \
redirect_port tcp 192.168.127.20:22 24

# No restrictions on LAN Interface
$add 00005 allow all from any to any via $lan

# No restrictions on OpenVPN Interface <ONLY IN PROD>
#$add 00006 allow all from any to any via $vpn

# No restrictions on Loopback Interface
$add 00010 allow all from any to any via lo0

# Reassemble inbound packets
$add 00099 reass all from any to any in

# NAT any inbound packets
$add 00100 nat 1 all from any to any in via vtnet0

#$add 00101 nat 1 all from any to any out via vtnet0

#Allow the packet through if it has an existing entry in the dynamic rules table
$add 00101 check-state

# Allow access to public DNS
# Repeat for each DNS server in /etc/resolv.conf

$add 00110 $skip tcp from any to 1.1.1.1 53 out via $pif setup keep-state
$add 00111 $skip udp from any to 1.1.1.1 53 out via $pif keep-state
$add 00112 $skip tcp from any to 8.8.8.8 53 out via $pif setup keep-state
$add 00113 $skip udp from any to 8.8.8.8 53 out via $pif keep-state

$add 00114 $skip tcp from any to any 53 out via $pif setup keep-state
$add 00115 $skip udp from any to any 53 out via $pif keep-state

# Allow access to ISP's DHCP server for cable/DSL configurations.
# Use the first rule and check log for IP address.
# Then, uncomment the second rule, input the IP address, and delete the first rule
# THIS IS USED TEMPORARILY
$add 00120 $skip log udp from any to any 67 out via $pif keep-state

# Allow all outbound connections
$add 00200 $skip all from any to any out via $pif setup keep-state
$add 00210 $skip icmp from any to any out via $pif keep-state

# Deny and log all other outbound connections  <DISABLED>
#$add 00299 deny log all from any to any out via $pif

# Deny all inbound traffic from non-routable reserved address spaces
#$add 00300 deny all from 192.168.0.0/16 to any in via $pif RFC 1918 private IP
#$add 00301 deny all from 172.16.0.0/12 to any in via $pif RFC 1918 private IP
#$add 00302 deny all from 10.0.0.0/8 to any in via $pif RFC 1918 private IP
$add 00303 deny all from 127.0.0.0/8 to any in via $pif #loopback
$add 00304 deny all from 0.0.0.0/8 to any in via $pif #loopback
$add 00305 deny all from 169.254.0.0/16 to any in via $pif #DHCP auto-config
$add 00306 deny all from 192.0.2.0/24 to any in via $pif #reserved for docs
$add 00307 deny all from 204.152.64.0/23 to any in via $pif #Sun cluster interconnect
$add 00308 deny all from 224.0.0.0/3 to any in via $pif #Class D & E multicast

# Deny public pings <DISABLED>
#$add 00310 deny icmp from any to any in via $pif

# Deny ident
$add 00315 deny tcp from any to any 113 in via $pif

# Deny all Netbios services.
$add 00320 deny tcp from any to any 137 in via $pif
$add 00321 deny tcp from any to any 138 in via $pif
$add 00322 deny tcp from any to any 139 in via $pif
$add 00323 deny tcp from any to any 81 in via $pif

#Deny fragments
$add 00330 deny all from any to any frag in via $pif

#Deny ACK packets that did not match the dynamic rule table
$add 00332 deny tcp from any to any established in via $pif

#Allow traffic from ISP's DHCP server.
#Replace (I won't as the static ip will be used in prod) x.x.x.x with the same IP address used in rule 00120.
$add 00360 allow udp from any to any 67 in via $pif keep-state

#Allow HTTP connections to internal web server <DISABLED>
#$add 00400 allow tcp from any to me 80 in via $pif setup limit src-addr 2

#Allow inbound SSH connections
$add 00410 allow tcp from any to me 22 in via $pif setup limit src-addr 2

#Allow inbound OpenVPN connections
$add 00415 allow udp from any to me 8480 in via $pif keep-state

#Reject and log all other incoming connections
$add 00499 deny log all from any to any in via $pif

#Everything else is denied and logged
$add 00999 deny log all from any to any

# Skipto location for outbound stateful rules
$add 01000 nat 1 log all from any to any out via $pif
$add 01001 allow log all from any to any
 
Just a long shot to check ... third last in the BUGS section at end of ipfw(8):

"Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your network traffic, please disable TSO on your NICs using ifconfig(8)."
 
Disabling TSO using ifconfig(8) as well as adding
Code: 
hw.vtnet.tso_disable=1
to /boot/loader.conf does not help. The behavior of the firewall does not change when I use these options.
 
fixinit75 said:
Disabling TSO using ifconfig(8) as well as adding
Code: 
hw.vtnet.tso_disable=1
to /boot/loader.conf does not help. The behavior of the firewall does not change when I use these options.
Click to expand...

Fair enough, just a stab in the dark - but leave them there. Good luck ...
 
Code: 
ipfw disable one_pass
ipfw nat 1 config log if vtnet0 reset unreg_only same_ports \
redirect_port tcp 192.168.127.20:22 24


Code: 
00100 allow ip from any to any via vtnet1
00150 allow ip from any to any via lo0
00200 reass ip from any to any in
00250 nat 1 ip from any to any in via vtnet0
00300 check-state :default
00310 allow tcp from any to me 22 setup keep-state :default
00450 skipto 1000 ip from any to 192.168.127.20 22 in via vtnet0 keep-state :default
00451 skipto 1000 ip from 192.168.127.0/24 to any out via vtnet0 keep-state :default
00500 skipto 1000 icmp from any to any out via vtnet0 keep-state :default
00550 deny ip from any to any
01000 nat 1 ip from any to any out via vtnet0
01050 allow ip from any to any
65535 deny ip from any to any
 
VladiBG said:
Code: 
ipfw disable one_pass
ipfw nat 1 config log if vtnet0 reset unreg_only same_ports \
redirect_port tcp 192.168.127.20:22 24


Code: 
00100 allow ip from any to any via vtnet1
00150 allow ip from any to any via lo0
00200 reass ip from any to any in
00250 nat 1 ip from any to any in via vtnet0
00300 check-state :default
00310 allow tcp from any to me 22 setup keep-state :default
00450 skipto 1000 ip from any to 192.168.127.20 22 in via vtnet0 keep-state :default
00451 skipto 1000 ip from 192.168.127.0/24 to any out via vtnet0 keep-state :default
00500 skipto 1000 icmp from any to any out via vtnet0 keep-state :default
00550 deny ip from any to any
01000 nat 1 ip from any to any out via vtnet0
01050 allow ip from any to any
65535 deny ip from any to any
Click to expand...
With this config the DNS does not work at all (ping google.com works neither from FreeBSD, nor from ubuntu).
When I try to connect to 192.168.127.0/24, the SYN packet is successfully received by ubuntu and it returns a SYN-ACK packet, but for some reason the connection does not establish, and the PC sends a retransmission of SYN packet, and the loop continues. I also can't connect to external ssh servers from ubuntu.
 
I have a question. But beforehand: I like this structured approach. I like that there is a mission statement (expectations), that there is a graphical representation (although I don't fully understand it), that there is logging in place, and a description of what works and what doesn't.

Now the question: if you do not know why those flows that are required. do not flow, then how can you ever be sure that such traffic as is unwelcome, will be reliably blocked?

I think the structured approach is the right way, but it needs to be taken further: a list of the desired flows, and then enabling one after another, figuring where it fails (with tcpdump and the ipfw logging), understanding why it fails - then refine, redo and tackle the next.
 
PMc said:
Now the question: if you do not know why those flows that are required. do not flow, then how can you ever be sure that such traffic as is unwelcome, will be reliably blocked?
Click to expand...
I am not sure that the unwelcome traffic will be blocked reliably, I only have facts that it is blocked, and can't predict the behavior of the firewall as I have never used it before.

PMc said:
I think the structured approach is the right way, but it needs to be taken further: a list of the desired flows, and then enabling one after another, figuring where it fails (with tcpdump and the ipfw logging), understanding why it fails - then refine, redo and tackle the next.
Click to expand...
I had tested the configuration before adding NAT to it, everything worked ok (except LAN connections to external networks as obviously they need the address translation in order to be able to establish connections outside the local network). All the problems start when I implement NAT. I don't really have much time to do further experiments as I am already over all deadlines. My main goal was to try fixing port redirects as the IPFW machine is expected to be used as a second WAN interface for a server, connected through OpenVPN, and clients are expected to connect through it. I have no success in my goal, and I can't install anything else like pfSense, because the iso mount function is broken on the hosting provider and they say it may take months to fix it.
 
fixinit75 said:
All the problems start when I implement NAT.
Click to expand...

Okay, I tried to walk that code a while - and the more I look at it the more beautiful it is. This is indeed done nicely, and from dry-running it I don't see an obvious error.

The only thing I can explain rightaway is this:
but the kernel logs show that the the sendto permission is denied for NTP requests
Click to expand...

Only ports 53 and 67 are allowed in the rules. 123 is not. (Rule 200 can only apply to TCP, because it requires the "setup" flag)
 
PMc said:
Only ports 53 and 67 are allowed in the rules. 123 is not. (Rule 200 can only apply to TCP, because it requires the "setup" flag)
Click to expand...
Dividing the rule 200 to separate TCP and UDP rules really solved the DNS problem. But, again, only for FreeBSD local connections.

What's the chance that there is a bug in the IPFW in-kernel NAT?
 
fixinit75 said:
What's the chance that there is a bug in the IPFW in-kernel NAT?
Click to expand...

100%.

Because I have found one. But that shouldn't bother you. It hits only successfully established connections, and manifests itself by occasional 10-15 minute stalls where NAT will not work for that connection. And it seems to concern only UDP (e.g. openvpn). (Search for the bug nr, I'm too tired now - my PSU decided to participate in the solstice celebrations, by exploding.)

If there is another bug, I fear one has to prove that. Like doing this and so get a log before/after where it gets clear what is (not) translated:

Code: 
00099 count log all from any to any in via vtnet0
00100 nat 1 all from any to any in via vtnet0
00101 count log all from any to any in via vtnet0
 
You must log in or register to reply here.
Back
Top