Solved ipfw + NAT mystery

[kumba]

So I just learned that there's two methods to doing NAT in FreeBSD. The apparently old natd + divert way, which is documented in the handbook, and the new in-kernel ipfw+nat way, that is randomly documented by Google. Is anyone ever going to update the handbook to over ipfw+NAT? The man page isn't exactly exhaustive on that topic.

Anyways, I'm at my wits' end over trying to get what should be a bog-simple NAT setup going for a wired LAN. For starters, my appliance has the following interfaces:

Interfaces: 6x GigE (em0 to em5), 1x ath0 via AR9462
WAN: em0 (SYNCDHCP)
LAN: bridge0 (em1 + em2)
WLAN: wlan0

For simplicity's sake, ignore the fact there's a WLAN interface. I've got hostapd and dhcpd all working on that interface and wifi devices connect and get an address. So once I get LAN NAT working, WLAN NAT should also just work. My ISP is a cable company, so the WAN is using SYNCDHCP to make sure there's an address assigned before the rest of the system tries to come up. Being that the interface ports are em*, I've already got all of the hardware bits, like TSO, LRO, TX/RX CSUM, etc, turned off. The bridge setup is currently using two em interfaces because I might use the other ports for span/mirroing traffic to a security appliance (not sure just yet).

I fought for a few hours with using natd + divert, but honestly didn't have a clue if I was doing that right. Then I stumbled across this post from 2017:

Solved - ipfw share internet

hello,i'am a newbie in ipfw, the scenario is this: i have a server that run 2 network interfaces,in the future it will be a squid proxy..but for now, step by step.... the lan is 192.168.50.0 (bce1) the wan if 200.xxx.xxx.xxx (bce0) so far i got this ipfw -f flush ipfw add 497 check-state...

forums.freebsd.org

That confused me at first, until I realized that there was another NAT capability.
I fiddled with it, but could not get that example to work. I then found an older post from 2015:

IPFW - IPFW Kernel NAT is not working

Please help. I have version 10.2-RELEASE with kernel NAT configured. Ping request pass to external adapter, but don't route back to internal. Internal (ue1): # tcpdump -ni ue1 | grep 5.255.255.5 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ue1...

forums.freebsd.org

Using that extremely simple script example, I could finally ping from my LAN out to the WAN and hit Google's quad-8 system. However, that's a terrible firewall. I tried to adapt some of the stuff from the first URL by having two NAT rules, putting check-state and the TCP/UDP/ICMP rules between the two NAT rules, and using "out xmit ${wan}". No dice.

Working script:

ipfw="/sbin/ipfw -q"
wan="em0"
lan="bridge0"
wifi="wlan0"

${ipfw} set disable 1
${ipfw} nat 1 config log if ${wan} deny_in same_ports unreg_only reset
${ipfw} add 0010 set 1 nat 1 ip from any to any via ${wan}
${ipfw} add 0020 set 1 reass all from any to any in
${ipfw} add 0030 set 1 check-state
${ipfw} add 0040 set 1 allow ip from any to any
${ipfw} set swap 0 1
${ipfw} delete set 1

The bit with using 'set' I got from fwbuilder.

Non-working script:

ipfw="/sbin/ipfw -q"
wan="em0"
lan="bridge0"
wifi="wlan0"

${ipfw} set disable 1
${ipfw} nat 1 config log if ${wan} deny_in same_ports unreg_only reset
${ipfw} add 0010 set 1 nat 1 ip from any to any in recv ${wan}
${ipfw} add 0020 set 1 reass all from any to any in
${ipfw} add 0030 set 1 check-state
${ipfw} add 0040 set 1 allow tcp from any to any out xmit ${wan} setup keep-state
${ipfw} add 0041 set 1 allow udp from any to any out xmit ${wan} keep-state
${ipfw} add 0042 set 1 allow icmp from any to any out xmit ${wan} keep-state
${ipfw} add 0050 set 1 nat 1 ip from any to any out xmit ${wan}
${ipfw} set swap 0 1
${ipfw} delete set 1

I also have

net.inet.ip.fw.one_pass=0

Which I think is the right setting? And IPv4 forwarding is also enabled. As far as I can tell, there's nothing else missing. It's some misunderstanding I have with the correct order of the rules, and correct rule directionality combined with getting the NAT rules right. If I can get the non-working script working, I think it'll be fairly easy to fix the actual firewall script I've put together. Said script denys all inbound and only allows select protocols outbound, plus some antispoofing rules. Nothing terribly complicated.

 

[VladiBG]

Your rule order is incorrect and you are missing skipto rules which are required when you split the NAT to inbound and outbound.

 

[driesm]

You can check the following thread for an example regarding IPFW in kernel NAT with stateful rules: https://forums.freebsd.org/threads/ipfw-nat-rule-being-skipped.56323/ obsigna's post.

You only need to switch off TSO for in-kernel NAT, this is a limitation of libalias. To do this system wide you can run sysctl net.inet.tcp.tso=0 and put it in /etc/sysctl.conf You'll also need the following: sysctl net.inet.ip.fw.one_pass=0, same here add it to /etc/sysctl.conf so that both are set when you boot the system.

Hope this helps.

 

[PMc]

So I just learned that there's two methods to doing NAT in FreeBSD. The apparently old natd + divert way, which is documented in the handbook, and the new in-kernel ipfw+nat way, that is randomly documented by Google. Is anyone ever going to update the handbook to over ipfw+NAT? The man page isn't exactly exhaustive on that topic.

Thats not the only problem - the worse thing is that the handbook example appears to contain errors - as we figured in some thread here. But then, the manpage is exhaustive, in the way manpages are supposed to be (declaring all the functionality - manpages are not tutorials).

I tried to adapt some of the stuff from the first URL by having two NAT rules, putting check-state and the TCP/UDP/ICMP rules between the two NAT rules, and using "out xmit ${wan}". No dice.

Such approach should work, if done correctly.

Non-working script:

${ipfw} add 0010 set 1 nat 1 ip from any to any in recv ${wan}
${ipfw} add 0020 set 1 reass all from any to any in
${ipfw} add 0030 set 1 check-state
${ipfw} add 0040 set 1 allow tcp from any to any out xmit ${wan} setup keep-state
${ipfw} add 0041 set 1 allow udp from any to any out xmit ${wan} keep-state
${ipfw} add 0042 set 1 allow icmp from any to any out xmit ${wan} keep-state
${ipfw} add 0050 set 1 nat 1 ip from any to any out xmit ${wan}

Lets see: the "recv wan" path appears to make sense: packets come in from the outer world, go to nat in line 10, so they get the internal quintuple on the packet. Then we reach check-state in 30. So if there is a dynamic rule active, the packet gets treated by it.
But the outgoing pachets "xmit wan" path do reach rule 40-42, here a dynamic rule gets installed ("keep-state"), and they are allowed. They will never reach the NAT in line 50, so NAT will just not work.
Furthermore, this ruleset only treats the WAN interface. All other packets on all other interfaces will nevertheless also go thru the ruleset, and will probably fall thru to the default rule 65535.
Furthermore, the check-state in line 30 will also treat all these packets on the various interfaces, which may or may not make sense.

Nothing terribly complicated.

The bad news is: if you want nat and stateful rules, it gets a bit complicated. Because, at the point where you do the "keep-state", you cannot do the "allow", because if you do "allow", then that packet leaves the system and does not reach the NAT.
There are a couple of ways to workaround this, but sadly, none of them is all too easy.

This one here

You can check the following thread for an example regarding IPFW in kernel NAT with stateful rules: https://forums.freebsd.org/threads/ipfw-nat-rule-being-skipped.56323/ obsigna's post.

appears to me the most compact, elegant and efficient approach, as long as the topology stays that simple.

For cases where this is not sufficient, e.g. when only part of our data shall go thru the NAT, or when we need source-based-routing or things like that, I have tried to solve the issue once-and-for-all, by sacrificing performance in favor of a generalized and (hopefully) logically correct rule building scheme, making the whole thing maintainable in the first place.
A rough prototype of that approach -most likely still full of bugs and errors- currently sits on oper.dinoex.de/flowm5 and it should spit out a rule list after entering the usual culprits (interfaces, flows, NATs, etc.). Some sparse in-line docs is also present.
No kind of express or implied warranties of any sort, as usual.

 

[kumba]

Your rule order is incorrect and you are missing skipto rules which are required when you split the NAT to inbound and outbound.

Ah, my bad, I meant to include the skipto bits in my "non-working" example. I was re-writing those three from memory.

You can check the following thread for an example regarding IPFW in kernel NAT with stateful rules: https://forums.freebsd.org/threads/ipfw-nat-rule-being-skipped.56323/ obsigna's post.

You only need to switch off TSO for in-kernel NAT, this is a limitation of libalias. To do this system wide you can run sysctl net.inet.tcp.tso=0 and put it in /etc/sysctl.conf You'll also need the following: sysctl net.inet.ip.fw.one_pass=0, same here add it to /etc/sysctl.conf so that both are set when you boot the system.

Hope this helps.

Okay, that link finally got me something that works, thanks! I've managed to adjust it based on my original firewall design, and it seems my chief error was I was using the old methodology of the final rule being a deny-all. Because I thought the outbound NAT rule would handle the routing of NAT'ed packets leaving, and thus a final deny would catch any weird stuff that fell through. Once I switched that over to allow, everything started working, and my understanding is, the outbound NAT rule just "mangles" the packet, while the final rule lets it leave the network.

Kind of a weird way to have to think about it, and I suspect very unique to IPFW, but it works.

 

[kumba]

So now I thought I'd try to get the WLAN piece running, since this appliance is also a wireless access point. I've got all of the WLAN bits setup, including hostapd, dhcpd, etc and clients can connect and get an address assigned. However, I am stumped on what I need to do to get packets on the WLAN, which use the 172.16.0.0/12 RFC1918 space, to route out through the WAN port. Basically, I'm assuming the wired LAN and WLAN are just like two separate subnets. Yet, the wired LAN just seems to always work, and the WLAN doesn't.

I don't want to bridge the WLAN into the LAN, because I want to keep those subnets separate. NAT'ing from the 172.16.0.0/12 space to the 192.168.0.0/16 space to get out seems insane. I suspect I am missing something simple, but w/o knowing what I should call that missing thing, I'm at a loss of what to search for. I've already added similar lines to ipfw to allow any traffic on the WLAN (hostapd has ap_isolate=1, so no worrying about wifi clients poking each other), and nothing appears wrong, except that one wifi client I am testing with can't reach the internet.

PS, when does the moderator-approval-for-new-posts thing get lifted? It's really annoying.

 

[longimanus]

PMc the link you suggested brings up a 403?

 

[PMc]

Thats cool! I'm now trying to figure out how one can achieve that...

What I'm seeing is: access to proxy failed, reason: SSL connection required
But that shouldn't be the case... What certainly is true: You should only be able to access the thing on https:// - and you will also get a "certificate invalid" or "authority unknown" (because I won't buy a commercial certificate for my prototype playground).

Oh yes - if You try to connect on unencrypted http, You indeed get 403 - this is not caught or auto-redirected. I thought nobody uses ordinary http anymore...

Addendum: someone managed to trigger a forgery protection false positive. Figuring out why this happens... (I know why I don't want to be a web developer. Had hoped that stuff could just be used.)

 

[obsigna]

... I've already added similar lines to ipfw to allow any traffic on the WLAN ...

Did you mean really ADD (after everything else) or did you INSERT the WLAN-allow-rule before the incoming NAT rule, where it would need to go to? In your original post, you did not specify any rule numbers, so I suspect this might be a problem with the rule order. The order of rule evaluation is strictly determined by rule numbers. You want to check this with the following command:
ipfw -d show

 

[PMc]

I've managed to adjust it based on my original firewall design, and it seems my chief error was I was using the old methodology of the final rule being a deny-all.

I might recommend to stay with that final rule doing a deny-all, and then adding an explicit allow rule where it is needed, e.g. after the NAT rule.
Then, for testing, you can temporarily make that allow rule to do logging, and watch /var/log/security (I suppose) for what is getting thru - and that, together with tcpdump, helps a lot.

 

[kumba]

...But that shouldn't be the case... What certainly is true: You should only be able to access the thing on https:// - and you will also get a "certificate invalid" or "authority unknown" (because I won't buy a commercial certificate for my prototype playground)...

You can always get a free certificate from Let's Encrypt. I think the only thing they don't offer are code-signing certs for free.

I might recommend to stay with that final rule doing a deny-all, and then adding an explicit allow rule where it is needed, e.g. after the NAT rule.
Then, for testing, you can temporarily make that allow rule to do logging, and watch /var/log/security (I suppose) for what is getting thru - and that, together with tcpdump, helps a lot.

That's a good point. I think I am going to enjoy having an "ipfw0" interface to attach tcpdump to. For years, I was always getting these outside nmap scans that I tracked via my old firewall because I had specific rules setup for each of nmap's scan types, including a Maimon scan (which was designed to single out BSD-powered devices). The way OpenWRT logged stuff, though, while I had a handy rule prefix I could look up in the syslog, I never tried to bother with configuring tcpdump to capture only rejected packets. ipfw0 on FreeBSD makes that trivial. Only thing I'll honestly miss are the rule log prefixes (unless I missed something in the ipfw(8) man page).

Did you mean really ADD (after everything else) or did you INSERT the WLAN-allow-rule before the incoming NAT rule, where it would need to go to? In your original post, you did not specify any rule numbers, so I suspect this might be a problem with the rule order. The order of rule evaluation is strictly determined by rule numbers. You want to check this with the following command:
ipfw -d show

Okay, I figured it out. PEBKAC. I had forgotten to allow access from the 172.16.x.x block I am using on the WLAN to talk to my DNS server on the 192.168.x.x LAN block. Had to update Unbound's config file to fix that, plus that machine's local firewall. The wireless client I was testing was an Android phone, which always tries connecting to Google-owned domains, and failing that, claims there's no internet access. Once I fixed that, holy cow do Android phones bombard DNS services with lookups once you take them out of airplane mode. I can clean that up later with some selective blocks. For now, everything appears to work, so now it's time to go clean config files up, turn off some debug logging, and commit my conf file changes to git so I keep a history of everything.

 

[PMc]

That's a good point. I think I am going to enjoy having an "ipfw0" interface to attach tcpdump to. For years, I was always getting these outside nmap scans that I tracked via my old firewall because I had specific rules setup for each of nmap's scan types, including a Maimon scan (which was designed to single out BSD-powered devices). The way OpenWRT logged stuff, though, while I had a handy rule prefix I could look up in the syslog, I never tried to bother with configuring tcpdump to capture only rejected packets. ipfw0 on FreeBSD makes that trivial. Only thing I'll honestly miss are the rule log prefixes (unless I missed something in the ipfw(8) man page).

I honestly don't know about "ipfw0".
The usual means of logging with ipfw is to syslogd(8). Any rule can have the "log" keyword, and then a message will be delivered via syslogd when the rule matches:

RULE FORMAT
The format of firewall rules is the following:

[rule_number] [set set_number] [prob match_probability] action
[log [logamount number]] [altq queue] [{tag | untag} number] body

Engaging some proper tcpdump on the respective incoming interface and watching that in parallel just makes the debugging easier - or at least that's how I am doing it.

 

[kumba]

I honestly don't know about "ipfw0".
The usual means of logging with ipfw is to syslogd(8). Any rule can have the "log" keyword, and then a message will be delivered via syslogd when the rule matches:

Engaging some proper tcpdump on the respective incoming interface and watching that in parallel just makes the debugging easier - or at least that's how I am doing it.

Add:

firewall_logif="YES"

To your /etc/rc.conf, restart networking, and you'll get an "ipfw0" interface that you can attach tcpdump do. All packets matching a "log" rule get a copy redirected to that interface. Attach tcpdump to it, and you only see the logged packets, not everything on the interface. It's pretty handy.

 

[kumba]

Okay, I more or less think that my main issue has been solved. I've had things running for about 24hrs now with no real issues. LAN and WLAN route packets fine now. Having tcpdump attached to the ipfw0 interface, I am getting a lot of dropped packets where, I guess, the dynamic rules for a particular session expires before the end of TIME_WAIT, thus the remote end sends back a FIN+ACK, or PSH+ACK, and gets blocked by the firewall. I suspect some tuning in sysctl can fix that. If anyone has pointers on what tunables to mess with, I am all ears.

Other than that, I thought I'd share a scrubbed copy of my script, where I've removed some identifying network address info and hope it helps someone out. If anyone sees any issues or knows of ways to better optimize it, please share.

Couple of questions first:

• Do "setup" and "keep-state" apply to SCTP as they do to TCP?
• Does anyone know if the issue(s) identified in PR201590 have been fixed, or should I set "net.inet.ip.fw.dyn_keepalive=0"?

Some quick notes:

• IPv4-only. There's actually a rule to block IPv6 at the WAN. I'll set that up eventually, once I learn more about NPTv6.
• Vars at the top need defining for IP addresses and interfaces (except lo0) for the script to work right.
• The table "ip4-rfc1918" needs your specific private IP space removed from it, lest you block all local traffic.
• The script is designed to replace /etc/rc.firewall. So remove "firewall_type" from rc.conf and add "firewall_script" instead, w/ the full path of this script.

#!/bin/sh -

# ipfw command.
ipfw="/sbin/ipfw -q"

# interfaces
lan="xxx0"
loop="lo0"
wan="yyy0"
wlan="zzz0"

# services
dns="xxx.xxx.xxx.xxx"
ntp="xxx.xxx.xxx.xxx"

# Copied from /etc/rc.firewall
#
# FreeBSD: Suck in the configuration variables.
if [ -z "${source_rc_confs_defined}" ]; then
    if [ -r /etc/defaults/rc.conf ]; then
        . /etc/defaults/rc.conf
        source_rc_confs
    elif [ -r /etc/rc.conf ]; then
        . /etc/rc.conf
    fi
fi

# FreeBSD: Setup loopback interface.
setup_loopback() {
    # Allow loopback traffic on loopback and deny everywhere else.
    ${ipfw} add 0001 set 1 allow all from any to any via ${loop}
    ${ipfw} add 0002 set 1 deny all from any to 127.0.0.0/8
    ${ipfw} add 0003 set 1 deny ip from 127.0.0.0/8 to any
    if [ $ipv6_available -eq 0 ]; then
        ${ipfw} add 0004 set 1 deny all from any to ::1
        ${ipfw} add 0005 set 1 deny all from ::1 to any
    fi
}

# FreeBSD: Setup IPv6.
setup_ipv6_mandatory() {
    [ $ipv6_available -eq 0 ] || return 0

    # Allow IPv6 DAD.
    ${ipfw} add 0006 set 1 allow ipv6-icmp from :: to ff02::/16

    # Allow IPv6 RS, RA, NS, NA, & redirect.
    ${ipfw} add 0007 set 1 allow ipv6-icmp from fe80::/10 to fe80::/10
    ${ipfw} add 0008 set 1 allow ipv6-icmp from fe80::/10 to ff02::/16

    # Allow ICMPv6 destination unreachable, NS, NA, & too-big.
    ${ipfw} add 0009 set 1 allow ipv6-icmp from any to any icmp6types 1,2,135,136
}

# Configure all rules in a shadow ruleset to avoid issues with the current-active ruleset.
${ipfw} set disable 1

# FreeBSD: Cribbed from /etc/rc.firewall
. /etc/rc.subr
. /etc/network.subr
afexists inet6
ipv6_available=$?

# FreeBSD: Loopback/IPv6 setup.
setup_loopback
setup_ipv6_mandatory

# Configure in-kernel NAT.  The NAT number is arbitrary.
${ipfw} nat 42 config log if ${wan} deny_in same_ports unreg_only reset

# IANA IPv4 RFC1918 addrs.  Remove whichever ones are used on your local
# network(s).
${ipfw} set 1 table ip4-rfc1918 flush
${ipfw} set 1 table ip4-rfc1918 add 10.0.0.0/8      # RFC1918 Class A
${ipfw} set 1 table ip4-rfc1918 add 172.16.0.0/12   # RFC1918 Class B
${ipfw} set 1 table ip4-rfc1918 add 192.168.0.0/16  # RFC1918 Class C

# IANA IPv4 special-use addrs.
${ipfw} set 1 table ip4-special flush
${ipfw} set 1 table ip4-special add 44.0.0.0/8      # Amateur Radio
${ipfw} set 1 table ip4-special add 100.64.0.0/10   # Shared
${ipfw} set 1 table ip4-special add 169.254.0.0/16  # APIPA
${ipfw} set 1 table ip4-special add 192.0.0.0/24    # IANA IPv4
${ipfw} set 1 table ip4-special add 192.0.2.0/24    # TEST-NET-1
${ipfw} set 1 table ip4-special add 192.88.99.0/24  # 6-to-4
${ipfw} set 1 table ip4-special add 198.18.0.0/15   # Benchmark
${ipfw} set 1 table ip4-special add 198.51.100.0/24 # TEST-NET-2
${ipfw} set 1 table ip4-special add 203.0.113.0/24  # TEST-NET-3
${ipfw} set 1 table ip4-special add 224.0.0.0/4     # Class D (remove if you use this)
${ipfw} set 1 table ip4-special add 240.0.0.0/4     # Class E


# ================================ LAN Rules =================================

# Block IPv4 special-use addresses on LAN.
${ipfw} add 0010 set 1 drop log ip4 from any to "table(ip4-rfc1918)" via ${lan}
${ipfw} add 0011 set 1 drop log ip4 from any to "table(ip4-special)" via ${lan}

# Allow all other traffic on the LAN.
${ipfw} add 0012 set 1 allow all from any to any via ${lan}


# =============================== WLAN Rules =================================

# Block IPv4 special-use addresses on WLAN.
${ipfw} add 0020 set 1 drop log ip4 from any to "table(ip4-rfc1918)" via ${wlan}
${ipfw} add 0021 set 1 drop log ip4 from any to "table(ip4-special)" via ${wlan}

# Allow all other traffic on the WLAN.
${ipfw} add 0022 set 1 allow all from any to any via ${wlan}


# ============================ WAN Defense Rules =============================

# Anti-spoofing rules using IPFW-specific capabilities.
${ipfw} add 0030 set 1 deny log ip from any to any not antispoof in recv ${wan}

# Block various uncommon IP option scan types.
${ipfw} add 0031 set 1 deny log all from any to any ipoptions lsrr,ssrr,rr,ts in recv ${wan}

# Block IPv4 special-use addresses on WAN.
${ipfw} add 0032 set 1 drop log ip4 from any to "table(ip4-special)" in via ${wan}

# Block SCTP on the WAN.
${ipfw} add 0033 set 1 abort log sctp from any to any via ${wan}

# Block several TCP scan types that are utterly invalid and thus can be
# checked before the dynamic rules are checked.
${ipfw} add 0034 set 1 unreach net-prohib log tcp from any to any tcpflags !syn,!fin,!ack,!psh,!rst,!urg in recv ${wan}   # NULL scan
${ipfw} add 0034 set 1 unreach net-prohib log tcp from any to any tcpflags !syn,fin,!ack,psh,!rst,urg in recv ${wan}      # Xmas scan
${ipfw} add 0034 set 1 unreach net-prohib log tcp from any to any tcpflags syn,fin,ack,psh,rst,urg in recv ${wan}         # Xmas Full scan

# Block IPv6 on the WAN for now.  This firewall script is IPv4-only for now.
${ipfw} add 0035 set 1 unreach6 admin-prohib log ip6 from any to any via ${wan}


# ============================ NAT Inbound Rules =============================

# Turn on packet reassembly for inbound traffic on the WAN
${ipfw} add 0040 set 1 reass all from any to any in recv ${wan}

# Handle Inbound NAT.  Outbound is further down.
${ipfw} add 0041 set 1 nat 42 ip4 from any to any in recv ${wan}

# Route established connections to the dynamic rules.
${ipfw} add 0042 set 1 check-state

# Block several TCP scan types that have to come //after// the check-state to
# avoid problems w/ existing dynamic rules.
${ipfw} add 0043 set 1 unreach net-prohib log tcp from any to any tcpflags syn,!fin,!ack,!psh,!rst,!urg in recv ${wan}    # SYN scan
${ipfw} add 0043 set 1 unreach net-prohib log tcp from any to any tcpflags !syn,!fin,ack,!psh,!rst,!urg in recv ${wan}    # ACK/Window scan
${ipfw} add 0043 set 1 unreach net-prohib log tcp from any to any tcpflags !syn,fin,!ack,!psh,!rst,!urg in recv ${wan}    # FIN scan
${ipfw} add 0043 set 1 unreach net-prohib log tcp from any to any tcpflags !syn,fin,ack,!psh,!rst,!urg in recv ${wan}     # Maimon scan


# ============================ Outbound Services =============================

# Allow all ICMP outbound, and only ICMP Type 8 inbound.
${ipfw} add 0050 set 1 skipto 1000 icmp from any to any out xmit ${wan} keep-state
${ipfw} add 0051 set 1 allow icmp from any to me icmptypes 8 in recv ${wan} keep-state

# Allow specific outbound for DNS, NTP
${ipfw} add 0052 set 1 skipto 1000 tcp from ${dns} to any 53 out xmit ${wan} setup keep-state
${ipfw} add 0053 set 1 skipto 1000 udp from ${dns} to any 53 out xmit ${wan} keep-state
${ipfw} add 0054 set 1 skipto 1000 udp from ${ntp} to any 123 out xmit ${wan} keep-state

# Deny outbound DNS, NTP from anything else.
${ipfw} add 0055 set 1 deny ip from not ${dns} to any 53 out xmit ${wan}
${ipfw} add 0056 set 1 deny ip from not ${ntp} to any 123 out xmit ${wan}

# Allow all other traffic.
${ipfw} add 0060 set 1 skipto 1000 tcp from any to any out xmit ${wan} setup keep-state
${ipfw} add 0061 set 1 skipto 1000 udp from any to any out xmit ${wan} keep-state

# Catch any rogue TCP/UDP packets.
${ipfw} add 0062 set 1 unreach 3 log tcp from any to any via ${wan}
${ipfw} add 0063 set 1 unreach 3 log udp from any to any via ${wan}


# ============================ NAT Outbound Rules ============================

# Handle Outbound NAT.  Inbound is further up.
${ipfw} add 1000 set 1 nat 42 ip4 from any to any out xmit ${wan}


# ============================================================================

# Rule #65535 defaults to accept any.

# Swap in the new ruleset.
${ipfw} set swap 0 1
${ipfw} delete set 1

 

[PMc]

• Do "setup" and "keep-state" apply to SCTP as they do to TCP?

Sorry, never used that.

• Does anyone know if the issue(s) identified in PR201590 have been fixed, or should I set "net.inet.ip.fw.dyn_keepalive=0"?

I don't think it is a bug to be fixed. Rather something that needs to be treated by extra rules.

net.inet.ip.fw.dyn_keepalive should be decided by the needs of the application(s). When we have keep-state on TCP sessions, and the session is open, the dynamic rule will have a 300 sec. timer. But there is no requirement for the TCP session to have any traffic - so if it just sits there and idles, after 300 sec. the dynamic rule gets reaped and the session is lost. Now all depends on what the application does in that case, when it tries to talk again. Some will drop the session and open a new one, some will sit there and fail utterly.

In the latter case we need the dyn_keepalive=1.

What I have found is that these TCP keepalive packets do appear on lo0 (incoming), which is not normal: a packet that originates locally and goes out onto a network will normally not appear incoming, only outgoing on the destination interface (and a packet going thru a gateway will appear incoming on one interface and outgoing on the other).

So, the people discussing that PR201590 where almost right: in comment #6 they blame the rule 300, while in fact it might rather be rule 200 that allows anything via lo0 not only incoming but also outgoing.

Correction: I did some further tests. The Comment #6 is right as far as gateway traffic (remote-remote) is concerned - then the rule 300 is the problem. Only In the other case (remote-local) keepalive packets appear "in via lo0" and rule 200 might allow them before NAT (didn't test that). Anyway:

This is not a bug to be fixed, it is something that needs a more precise design of the rules: try for instance:

00200 allow ip from any to any recv lo0 in
00201 allow ip from any to any xmit lo0 out

But, as also is mentioned in the comment #6, it is doubtful if that example is best practise, anyway.

 

[kumba]

Sorry, never used that.

SCTP is a stateful transport layer protocol developed by the SIGTRAN group for telephony stuff. FreeBSD actually had one of the earliest implementations of it. I noticed a reference to it in the man page for ipfw by dedicated "abort" and "abort6" actions to deal with rogue probes for SCTP (e.g., nmap has two scan types for it). Hence it made me wonder if the ipfw support for it included the stateful elements.

I don't think it is a bug to be fixed. Rather something that needs to be treated by extra rules.

net.inet.ip.fw.dyn_keepalive should be decided by the needs of the application(s). When we have keep-state on TCP sessions, and the session is open, the dynamic rule will have a 300 sec. timer. But there is no requirement for the TCP session to have any traffic - so if it just sits there and idles, after 300 sec. the dynamic rule gets reaped and the session is lost. Now all depends on what the application does in that case, when it tries to talk again. Some will drop the session and open a new one, some will sit there and fail utterly.

In the latter case we need the dyn_keepalive=1.

What I have found is that these TCP keepalive packets do appear on lo0 (incoming), which is not normal: a packet that originates locally and goes out onto a network will normally not appear incoming, only outgoing on the destination interface (and a packet going thru a gateway will appear incoming on one interface and outgoing on the other).

So, the people discussing that PR201590 where almost right: in comment #6 they blame the rule 300, while in fact it might rather be rule 200 that allows anything via lo0 not only incoming but also outgoing.

Correction: I did some further tests. The Comment #6 is right as far as gateway traffic (remote-remote) is concerned - then the rule 300 is the problem. Only In the other case (remote-local) keepalive packets appear "in via lo0" and rule 200 might allow them before NAT (didn't test that). Anyway:

This is not a bug to be fixed, it is something that needs a more precise design of the rules: try for instance:

00200 allow ip from any to any recv lo0 in
00201 allow ip from any to any xmit lo0 out

But, as also is mentioned in the comment #6, it is doubtful if that example is best practise, anyway.

I think the rules to set loopback up should be fine. I actually copied the "setup_loopback" function from rc.firewall, so if that's not right, then it's an issue that needs a bug of its own to be filed.

So that, I guess, leaves the question of the way one currently handles traffic on the internal LAN interface. If avoiding the use of "via" seems to be the issue, and passing inbound LAN traffic early is fine, then for the stateful NAT case, wouldn't one want to use skipto to jump to the outbound NAT rule?:

${ipfw} add 0200 allow ip from any to any in recv ${lan}
${ipfw} add 0201 skipto 1000 ip4 from any to any out xmit ${lan}
...
${ipfw} add 1000 set 1 nat 1 ip4 from any to any out xmit ${wan}

I put "ip4" on that outbound LAN rule, as for ip6 traffic, there probably needs to be an extra rule to simply allow ip6 traffic on the LAN to just pass. Or would the NAT rule just ignore IPv6 and let it fall through to the final 65535 rule?

 

[TimoG]

I might be wrong about this, but I think you have to insert a rule that allows passing the packet out via the WAN interface after the NAT rule.

In my ipfw firewall, the last couple of lines look like this:

32000 nat 1 ip4 from any to any out via ${wan}
32100 allow ip from any to any via ${wan}

Rule 32100 takes care of IPV6 packets as well, but if I omit it both the nat'd ipv4 packet and the ipv6 packets hit the implicit deny rule. IPV6 packets definitely don't match the nat rule, so they'll likely hit the implicit deny rule without further ado.

 

[kumba]

I might be wrong about this, but I think you have to insert a rule that allows passing the packet out via the WAN interface after the NAT rule.

I'm relying on the final rule, 65535, to allow that to happen, since I also set the loader tunable "net.inet.ip.fw.default_to_accept" to 1. Once I get more comfortable with using an ipfw ruleset, I might revert that back to a default deny rule and add my own catch-all as 65534, as someone else suggested on these forums.