IPFW Very simple IPFW firewall w/ kernel NAT not reliable, similar pf firewall works fine

TimoG

New Member


Messages: 5

This issue has had me stumped for month or two now. I've been running a home-built router in front of my cable modem that is running 11.2-stable. -stable because I hit an issue with console initialisation that IIRC had been patched in -stable but not -release at the point when I last rebuilt the box.

This firewall box has been running a simple pf firewall configuration for a long time with no noticable hiccups. The pf config is as follows:

Code:
ext_if = "em0"
int_if = "{ em1, em2, em2 }"

localnet = "{ em1:network, em2:network, em3:network }"

set skip on lo0

scrub in all

nat on $ext_if from $localnet to any -> ($ext_if)

block in
pass out

pass quick on $int_if no state
antispoof quick for $int_if

pass from { lo, em1:network, em2:network, em3:network } to any keep state
I then tried to duplicate the above pf configuration and translated it to ipfw as the first step:

Code:
lan_if="em1, em2, em3"
wan_if="em0"
#netbios="137,138,139"
setup_loopback() {
        ############
        # Only in rare cases do you want to change these rules
        #
        ${fwcmd} add pass all from any to any via lo0
        ${fwcmd} add deny all from any to 127.0.0.0/8
        ${fwcmd} add deny ip from 127.0.0.0/8 to any
        if [ $ipv6_available -eq 0 ]; then
                ${fwcmd} add deny all from any to ::1
                ${fwcmd} add deny all from ::1 to any
        fi
}

${fwcmd} -f flush
${fwcmd} nat 1 config if ${wan_if} same_ports reset
${fwcmd} add reass all from any to any in

# Throw in default configurations for loopback

setup_loopback

${fwcmd} add allow ip from any to any via em1
${fwcmd} add allow ip from any to any via em2
${fwcmd} add allow ip from any to any via em3

${fwcmd} add deny ip from any to any not verrevpath in

${fwcmd} add nat 1 ip from any to any via ${wan_if} in
${fwcmd} add check-state
${fwcmd} add deny log tcp from any to any established via ${wan_if}

${fwcmd} add skipto 32000 tcp from any to any via ${wan_if} out setup keep-state
${fwcmd} add skipto 32000 udp from any to any via ${wan_if} out keep-state

${fwcmd} add pass icmp from any to any

${fwcmd} add deny tcp from any to any via ${wan_if}
${fwcmd} add deny udp from any to any via ${wan_if}

${fwcmd} add 32000 nat 1 ip from any to any via ${wan_if} out
On the face of it, the ipfw configuration works. It looks like it nats correctly and I can see the expected firewall rules being dynamically generated as connections are being opened. However the firewall isn't reliable at that point and seems to be randomly deciding to establish connections (mostly http/s) or not. A typical example is that I'm browsing a forum and switching to the next page will require multiple reloads until the connection is finally established. I haven't wireshark'd the traffic between the router and the cable modem yet but to me this looks very much like the responses from outside servers are either not being properly nat'd back or are filtered out by firewall rules that shouldn't be filtering them out. Commenting out the line
Code:
${fwcmd} add deny log tcp from any to any established via ${wan_if}
improves matters somewhat, but doesn't fix the issue. This leads me to believe the issue is the in-kernel NAT, not the firewall rules themselves.

Searching the forum brought up a couple of posts discussing that TSO doesn't play nicely with ipfw's kernel NAT. ifconfig's output shows that TSO is not enabled on any of the interfaces on the system (em0 being the WAN facing interface):

Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: no carrier
I've got the distinct feeling that I'm missing something blindingly obvious here, but I don't seem to be able to figure out exactly where the misconfiguration resides.
 

obsigna

Aspiring Daemon

Reaction score: 490
Messages: 859

What happens, when you add to your rule number 32000 the specifier out, so the whole rule would read:

${fwcmd} add 32000 nat 1 ip from any to any via ${wan_if} out

Also verify, that the sysctl(8) value of net.inet.ip.fw.one_pass is set to 0.
 
OP
OP
T

TimoG

New Member


Messages: 5

VladiBG - that line is there to catch connections that pretend to be established based on the header flags, but aren't recognized by the stateful firewall (that's why there is a check-state right above it). The rule gets hit surprisingly often, which either suggests that there is something wrong with my firewall rules, or that I'm getting a bunch of junk pretend connections. That said, commenting it out or removing it unfortunately doesn't seem to fix the issue.

obsigna - rule 32000 actually has 'out' as part of the rule. That was a copy&paste error. It looks like net.inet.ip.fw.one_pass is set to 1 and setting it to 0 breaks the whole firewall. From reading up a bit on things, this suggests issues with the ruleset. I'll go and debug that one a bit more.
 

VladiBG

Aspiring Daemon

Reaction score: 209
Messages: 509

If you want to block only the packets with TCP flag ACK then use tcpflags instead of established.
The check-state execute the action from the rule which invoked the setup/state in your case this is skipto and after it i don't see any allow rule.

Here is a quick example of in-kernal nat with net.inet.ip.fw.one_pass=0 don't use it in production as it's created only for demonstration.

hn1 is LAN facing interface
hn0 is WAN

Code:
ipfw nat 1 config if hn0 log deny_in same_ports reset

00010 allow ip from any to any via lo0
00020 allow ip from any to any via hn1
00030 allow icmp from me to any keep-state
00040 allow udp from me to any keep-state
00050 allow ip from me to any setup keep-state
00100 nat 1 ip from any to any in via hn0
00200 check-state
00300 skipto 600 ip from 192.168.50.0/24 to any out via hn0 setup keep-state
00350 skipto 600 udp from 192.168.50.0/24 to any out via hn0 keep-state
00400 skipto 600 icmp from 192.168.50.0/24 to any out via hn0 keep-state
00500 deny ip from any to any via hn0
00600 nat 1 ip from any to any out via hn0
00700 allow ip from any to any
65535 deny ip from any to any
 
OP
OP
T

TimoG

New Member


Messages: 5

Thanks VladiBG. I experimented with my rules a bit more and arrived at a fairly similar ruleset, but one that was missing a few of the rules you added. I'll update mine based on your suggestions and will report back!
 

VladiBG

Aspiring Daemon

Reaction score: 209
Messages: 509

Instead of using dynamic states you can simplify the config and use only the alias nat table with deny_in option with net.inet.ip.fw.one_pass=1
The difference here is to be sure that deny_in option is present in the nat config otherwise you will permit all incoming traffic.

em0 is WAN
lo1 is LAN facing

Code:
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ip6 from any to any proto ipv6-icmp ip6 icmp6types 1
01000 allow ip6 from any to any proto ipv6-icmp ip6 icmp6types 2,135,136
01100 check-state :default
01200 allow tcp from me to any established
01300 allow tcp from me to any setup keep-state :default
01400 allow udp from me to any keep-state :default
01500 allow icmp from me to any keep-state :default
01600 allow ipv6-icmp from me to any keep-state :default
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 67 out
01800 allow udp from any 67 to me 68 in
01900 allow udp from any 67 to 255.255.255.255 68 in
02000 allow udp from fe80::/10 to me 546 in
02100 allow icmp from any to me icmptypes 8
02200 allow ip6 from any to me proto ipv6-icmp ip6 icmp6types 128,129
02300 allow icmp from any to me icmptypes 3,4,11
02400 allow ip6 from any to me proto ipv6-icmp ip6 icmp6types 3
02510 allow tcp from any to me 22
05000 allow ip from any to any via lo1
55000 nat 1 ip from any to any via em0
65535 deny ip from any to any
 
Top