ipfw nat problems

graudeejs · Jan 3, 2011

I'm trying to set up nat on my server and it works with simple & dumb rules, but I can't get it to work with normal rules.

My old setup used to be:

Code:

                            +---------------------------------+
          ++-------+        | (Router)                        |
          |        +--------+ HOME_NET_IP <->   192.168.128.1 |
ISP ------+ Switch |        +------------------------+--+--+--+
          |        +---+                             |  |  |
          +--------+   |                             |  |  +-- 192.168.128.4 (laptop2.pc)
                       | +------------------------+  |  +----- 192.168.128.3 (laptop.pc)
                       | | SERVER_IP1             |  |        +----------------------------+
                       +-+ SERVER_IP2             |  +--------+ 192.168.128.2 (desktop.pc) |
                         | SERVER_IP3 192.168.0.1 +-----------+ 192.168.0.2                |
                         |  (Server)              |           +----------------------------+
                         +------------------------+

The problem is that router is only 10Mbps, and after porer failures it has trouble to start serving clients. (eventually after few hors it just work)

So to lose the router I'd like to build my setup like this:

Code:

        +------------- ---------------+      +--------+
        | HOME_NET_IP <-> 192.168.0.1 + -----+        +----------- 192.168.0.2 (desktop.pc)
        | SERVER_IP1                  |      | Switch +----------- 192.168.0.3 (laptop.pc)
ISP ----+ SERVER_IP2                  |      |        +----------- 192.168.0.4 (laptop2.pc)
        | SERVER_IP3        (Server)  |      +--------+
        +----------------- -----------+

Benefit would also be 1Gbps LAN (I don't use wifi at home)

For this to work I've modified my ipfw rules on server.
Just to get me started adding

Code:

ipfw_cmd "3 add divert natd ip from any to any via $EIF"
ipfw_cmd "4 add allow ip from any to any via lo0"
ipfw_cmd "5 add allow ip from any to any via $IIF keep-state"
ipfw_cmd "8 add check-state"

Did the trick (IIF - Internal interface, EIF - external interface) (but obviously sux)

So I started reading manual and adopting example to my needs, but I can't get it to work.

I will post my ipfw rules in next post to avoid 10K character limit per post

graudeejs · Jan 3, 2011

Code:

#!/bin/sh

readonly IPFW='/sbin/ipfw'
ipfw_cmd() {
	$IPFW "-q $*" || echo "ERR: cmd failed: '$*'" >&2
}


readonly EIF='rl0'	# external interface
readonly IIF='re0'	# internal interface

readonly ROOT_IP='[red]xxx.xxx.xxx.xx1[/red]"
readonly MAIN_IP='[red]xxx.xxx.xxx.xx2[/red]'
readonly SHARE1_IP='[red]xxx.xxx.xxx.xxx3[/red]'

readonly JAIL_IPS="$MAIN_IP,$SHARE1_IP"
readonly ALL_IPS="$ROOT_IP,$JAIL_IPS"

readonly LAN_EXT_IP="[red]xxx.xxx.xxx.xx4[/red]"
readonly LAN_GATEWAY_IP="192.168.0.1"
readonly LAN_ADMIN_IP="192.168.0.2"
readonly LAN_IPS="192.168.0.0/24"


readonly RFC1918='10.0.0.0/8, 172.16.0.0/12, 129.168.0.0/16'

RULE_NUMBER=0

for i in $WTABLE_MAIL_MY $WTABLE $BTABLE $BTABLE_MAIL_MY $WTABLE_FTP $WTABLE_LV; do
	ipfw_cmd "table $i flush"
done
ipfw_cmd "flush"



ipfw_cmd "10 add deny log src-ip table($BTABLE) // blacklisted IPs"
ipfw_cmd "20 add deny log ip from any to any via ng0"

ipfw_cmd "40 add deny log ip from any to $RFC1918 via $EIF // RCF1918"
ipfw_cmd "50 add deny log ip from $RFC1918 to any via $EIF // RCF1918"

ipfw_cmd "60 add allow ip from any to any via lo0"
ipfw_cmd "70 add allow ip from any to any via $IIF"


ipfw_cmd "110 add divert natd ip from any to any in via $EIF"
ipfw_cmd "111 add check-state"
#ipfw_cmd "120 add skipto 10000 tcp from any to any http,https out via $EIF setup keep-state"
#ipfw_cmd "121 add skipto 10000 udp from any to any domain,nameserver out via $EIF setup keep-state"
ipfw_cmd "121 add skipto 10000 ip from any to any out via $EIF setup keep-state"
ipfw_cmd "122 add skipto 10000 icmp from any to any out via $EIF keep-state"


ipfw_cmd "9999 add deny log ip from any to any"
ipfw_cmd "10000 add divert natd ip from any to any in via $EIF"
ipfw_cmd "10001 add allow ip from any to any


################################################################################
################################################################################

[red]#SERVER RULES GOES HERE (doesn't go beyond 5K rule[/red]

# vim: set ts=4 sw=4:

I removed rules, that aren't relevant.

Anyone see anything wrong here?

graudeejs · Jan 3, 2011

Well I'll be dam.... Example in handbook doesn't seam to work as well.

btw, I didn't tell about my /etc/rc.conf

Code:

ifconfig_rl0="inet [b]LAN_PUBLIC_IP[/b]/24"
ifconfig_rl0_alias0="inet SERVER_IP1/32"
ifconfig_re0="inet 192.168.0.1/24"
defaultrouter="xxx.xxx.xxx.1"
firewall_enable="YES"
firewall_script="/etc/ipfw.sh"
gateway_enable="YES"
natd_enable="YES"
natd_interace="re0"
natd_flags="-m -a [b]LAN_PUBLIC_IP[/b]"

graudeejs · Jan 4, 2011

Do I need proxy to NAT https?

phoenix · Jan 4, 2011

No. NAT doesn't touch the contents of the packets, just the IP header to change the address.

graudeejs · Jan 4, 2011

Well I don't understand, why I can browse http, but can't get any https to load

Also It looks like there are some nat bugs in ipfw (in FreeBSD-8)
Heck I can't even reproduce example in handbook (doesn't work at all or, I'm doing something wrong. I improvise a bit, since I have server running as well) or manual (the very first command gives error)

Code:

# ipfw add nat 123 all from any to any
ipfw: getsockopt(IP_FW_ADD): Invalid argument

I think tonight I will try to setup nat with pf and see how well that goes.

phoenix · Jan 5, 2011

I haven't used the built-in nat support yet, but using natd is like so:

Code:

# natd -port 8668 -same_ports -use_sockets -alias_address <public_ip> -redirect_address <private_ip> <public_ip>

# ipfw add allow tcp from <private_ip> to any 443 in recv <private>
# ipfw add divert 8668 tcp from <private_ip> to any 443 out xmit <public>
# ipfw add allow tcp from <public_ip> to any 443

# ipfw add divert 8668 tcp from any 443 to <public_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> out xmit <private> established

Reading the man page, it looks like the equivalent should be along the lines of:

Code:

# ipfw nat 1 config ip <public_ip> same_ports

# ipfw add allow tcp from <private_ip> to any 443 in recv <private>
# ipfw add nat 1 tcp from <private_ip> to any 443 out xmit <public>
# ipfw add allow tcp from <public_ip> to any 443

# ipfw add nat 1 tcp from any 443 to <public_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> out xmit <private> established

graudeejs · Jan 5, 2011

phoenix said:

I haven't used the built-in nat support yet, but using natd is like so:

Code:

# natd -port 8668 -same_ports -use_sockets -alias_address <public_ip> -redirect_address <private_ip> <public_ip>

# ipfw add allow tcp from <private_ip> to any 443 in recv <private>
# ipfw add divert 8668 tcp from <private_ip> to any 443 out xmit <public>
# ipfw add allow tcp from <public_ip> to any 443

# ipfw add divert 8668 tcp from any 443 to <public_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> out xmit <private> established

-redirect_address <private_ip> <public_ip> means static NAT. I've read man page over and over and the way I understand is that only PC behind NAT having private_ip will receive traffic. But I have 3 PC's behind NAT. As I understand (from what I've read so far) I need dynamic NAT.

phoenix said:

Reading the man page, it looks like the equivalent should be along the lines of:

Code:

# ipfw nat 1 config ip <public_ip> same_ports

# ipfw add allow tcp from <private_ip> to any 443 in recv <private>
# ipfw add nat 1 tcp from <private_ip> to any 443 out xmit <public>
# ipfw add allow tcp from <public_ip> to any 443

# ipfw add nat 1 tcp from any 443 to <public_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> out xmit <private> established

Code:

# ipfw nat 1 config ip xxx.xxx.xxx.xxx same_ports
ipfw: setsockopt(IP_FW_NAT_CFG): Invalid argument

phoenix · Jan 5, 2011

Whoops, you're right. Too much time working with servers and 1:1 NAT.

Just remove the redirect_address part, leaving only -alias_address.

That error message means your ipfw doesn't support in-kernel NAT.

Have you loaded the ipfw_nat.ko module? Or compiled a custom kernel with IPFIREWALL_NAT included?

graudeejs · Jan 6, 2011

I had custom kernel, but I didn't have IPFIREWALL_NAT included.
Thanks for hint, I will tray later again.... (in few days, or weeks), when I have some free time again.

I will post about my success/failures.

qsecofr · Jan 6, 2011

Plz post the follow-up when you've found the successful syntax and/or kernel config for running nat built-in versus natd on a divert socket. I've tried a couple times and not had success. And the man page didn't help me understand fully.

I've got ipfw and natd running on FreeBSD-7.2 with http & https enabled. Not sure exactly what you're not getting from your firewall. Maybe a few example lines of denied packets from /var/log/security will help.

My setup looks pretty much like your second diagram, if a bit simpler. Your mileage may vary with established & http packets. Some excerpted rules may or may not help, but here goes:

server has 3 interfaces. 1 external. 2 internal: 1 wired, 1 wireless
oif defined in ipfw rule script = outside interface, static routable IP

Code:

if [ -r /etc/rc.conf ]; then
        . /etc/rc.conf
fi

..cut dummynet

Code:

$ipfw -q add allow all from any to any via lo0

..cut table for denied IPs

Code:

# every packet in or out.  from rc.conf
if natd_enable="YES";  then
        $ipfw -q add divert natd all from any to any via $oif
else
        $ipfw -q add nat 1 all from any to any via $oif
fi

..cut comments

Code:

# antispoof
$ipfw -q add deny log ip from any to any not antispoof in

$ipfw -q add check-state

# from rc.conf
# VPN to work seem to fragment some packets
if test "$FW_ALLOW_FRAGMENTS" = "YES"; then
        $ipfw -q add allow log all from any to any frag
fi

# control deny RST or ACK packets that didnt match dynamic table
# from rc.conf
if test "$FW_ALLOW_ESTABLISHED" = "YES"; then
        $ipfw -q add allow log all from any to any established out via $oif
        $ipfw -q add allow log all from any http,https to any established in via $oif
        $ipfw -q add allow all from any to any established not via $oif
fi

..cut other services

Code:

# http in
if test "$apache22_enable" = "YES";  then
        $ipfw -q add allow tcp from any to me http,https in setup keep-state
fi
# http out 1935 some sort of flash or media streaming
if test "$FW_ALLOW_HTTP" = "YES"; then
        $ipfw -q add allow tcp from any to not me http,https,8080,8888,1935 in not via $oif setup keep-state
        $ipfw -q add allow tcp from any to not me http,https,8080,8888,1935 out setup keep-state
fi

..cut etc etc

phoenix · Jan 6, 2011

You're missing the ipfw nat 1 config line for the non-natd version. IOW, it'll never work, because ipfw doesn't know how to mangle the packets.

graudeejs · Jan 9, 2011

I recompiled kernel with

Code:

options         IPFIREWALL
options         IPFIREWALL_FORWARD
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=5
options         IPDIVERT
options         IPFIREWALL_DEFAULT_TO_ACCEPT

[B]options         IPFIREWALL_NAT
options         LIBALIAS[/B]

phoenix said:

Reading the man page, it looks like the equivalent should be along the lines of:

Code:

# ipfw nat 1 config ip <public_ip> same_ports

# ipfw add allow tcp from <private_ip> to any 443 in recv <private>
# ipfw add nat 1 tcp from <private_ip> to any 443 out xmit <public>
# ipfw add allow tcp from <public_ip> to any 443

# ipfw add nat 1 tcp from any 443 to <public_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> in recv <public> established
# ipfw add allow tcp from any 443 to <private_ip> out xmit <private> established

Your rules got me lost... (digging manpage AGAIN)

graudeejs · Jan 9, 2011

Managed to get https also running with these rules and natd

Code:

readonly IPFW='/sbin/ipfw'
ipfw_cmd() {
    $IPFW "-q $*" || echo "ERR: cmd failed: '$*'" >&2
}

ipfw_cmd "add 10 divert natd ip from $LAN_IPS to any out via $EIF"
ipfw_cmd "add 11 divert natd ip from any to $LAN_EXT_IP in via $EIF"

ipfw_cmd "add 12 pass tcp from $LAN_EXT_IP to any ssh,smtp,http,pop3,https,8080 via $EIF"
ipfw_cmd "add 13 pass tcp from any ssh,smtp,http,pop3,https,8080 to $LAN_IPS via $EIF established"
ipfw_cmd "add 14 pass tcp from $LAN_IPS to any ssh,smtp,http,pop3,https,8080 via $IIF"
ipfw_cmd "add 15 pass tcp from any ssh,smtp,http,pop3,https,8080 to $LAN_IPS via $IIF established"
ipfw_cmd "add 16 pass all from any to any via $IIF"

phoenix · Jan 10, 2011

killasmurf86 said:
Your rules got me lost... (digging manpage AGAIN)

Configure the NAT instance that will convert private IPs to <public_ip>:

Code:

# ipfw nat 1 config ip <public_ip> same_ports

Allow traffic from stations on the LAN (<private_subnet>), coming in on the LAN NIC (<private>):

Code:

# ipfw add allow tcp from <private_subnet> to any 443 in recv <private>

NAT traffic from the <private_subnet>, as it leaves the <public> NIC:

Code:

# ipfw add nat 1 tcp from <private_subnet> to any 443 out xmit <public>

Allow the NAT'd traffic out the <public> NIC:

Code:

# ipfw add allow tcp from <public_subnet> to any 443 out xmit <public>

And then you do the reverse, for the traffic returning to the LAN.

NAT traffic coming back (established), as it enters the <public> NIC:

Code:

# ipfw add nat 1 tcp from any 443 to <public_subnet> in recv <public> established

Allow the NAT'd traffic:

Code:

# ipfw add allow tcp from any 443 to <private_subnet> in recv <public> established

Allow the NAT'd traffic out to the LAN:

Code:

# ipfw add allow tcp from any 443 to <private_subnet> out xmit <private> established