in Kernel NAT, IPFW, and ballanced forwarding

D

DarcyB

I'm attempting to mimic a setup in 8.x that I have currently running on a 4.x machine using natd and ipfw to provide load access to multiple (3) internet connections for systems behind the router. This setup works 100% as exptected with the current 4.x machine, however all the possible configurations I use for 8.x do not work as expected. The most common problems are either packets going out that are not nated, or else packets going out the wrong interface. I've tried both in kernel NAT and userland, Making use of setfib, or doing it via fwd rules (like was required in 4.x). I'm not sure if I'm just missing something or if it's truly broken. Before anybody suggests "use PF" I have multiple reasons for needing to stay with IPFW that I won't get into here.

Please see the following trimmed down ipfw config:

Code: 
00100 23136 1363628 allow ip from any to any via lo0
00150 21470 1293288 skipto 50000 tcp from any 1023-65535 to me dst-port 22
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
05000 22992 5108938 nat 1 ip from any to me in via sf0
05001 187 13879 nat 2 ip from any to me in via sf1
05002 115 17224 nat 3 ip from any to me in via sf3
05100 77 21743 prob 0.500000 setfib 1 ip from 192.168.0.0/16 to not 192.168.0.0/16 in via em0
05101 130 32006 skipto 5500 ip from any to any fib 1
05110 45 12933 prob 0.500000 setfib 2 ip from 192.168.0.0/16 to not 192.168.0.0/16 in via em0
05111 59 14304 skipto 5500 ip from any to any fib 2
05120 12 3317 prob 0.400000 setfib 3 ip from 192.168.0.0/16 to not 192.168.0.0/16 in via em0
05121 16 3650 skipto 5500 ip from any to any fib 3
05150 23 6102 setfib 1 ip from 192.168.0.0/16 to not 192.168.0.0/16 in via em0
05200 48101 7425881 skipto 6000 ip from 192.168.0.0/16 to 192.168.0.0/16
05201 24486 4397388 skipto 6000 ip from me to any
05500 153 38108 nat 1 ip from any to any fib 1
05510 59 14304 nat 2 ip from any to any fib 2
05520 16 3650 nat 3 ip from any to any fib 3
05521 26078 6021261 skipto 6000 ip from any to any
60000 1 400 fwd xxx.xxx.xxx.xxx ip from any to any fib 1
60001 59 14304 fwd yyy.yyy.yyy.yyy ip from any to any fib 2
60002 5 1200 fwd zzz.zzz.zzz.zzz ip from any to any fib 3
64000 119907 19081756 allow log logamount 1000000000 ip from any to any
65535 9 4404 deny ip from any to any


# ifconfig sf0
sf0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MU LTICAST> metric 0 mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
ether 00:00:d1:ee:57:49
inet xxx.xxx.xxx.180 netmask 0xfffff800 broadcast xxx.xxx.xxx.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active

#ifconfig sf1
sf1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MU LTICAST> metric 0 mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
ether 00:00:d1:ee:57:4a
inet yyy.yyy.yyy.250 netmask 0xfffffc00 broadcast yyy.yyy.yyy.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active

# ifconfig sf3
sf3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MU LTICAST> metric 0 mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
ether 00:00:d1:ee:57:4c
inet zzz.zzz.zzz.27 netmask 0xfffffc00 broadcast zzz.zzz.zzz.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active

and via tcpdump I see the following behavior:
Code: 
07:20:36.073159 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 1412, offset 0, flags [DF], 
proto TCP (6), length 60)
yyy.yyy.yyy.250.22 > 119.188.7.177.39256: Flags [S.], cksum 0x5b9a (incorrect -> 0x6684), seq 2784871141, ack 3990006922, win 65535, 
options [mss 1460,nop,wscale 3,sackOK,TS val 3658047670 ecr 563711227], length 0
07:26:08.215063 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 1466, offset 0, flags [DF], 
proto TCP (6), length 40)
yyy.yyy.yyy.250.445 > 174.4.26.20.4427: Flags [R.], cksum 0xa431 (incorrect -> 0x3f22), seq 0, ack 1678660993, win 0, length 0
07:26:11.182044 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 1468, offset 0, flags [DF], 
proto TCP (6), length 40)
yyy.yyy.yyy.250.445 > 174.4.26.20.4427: Flags [R.], cksum 0xa431 (incorrect -> 0x3f22), seq 0, ack 1, win 0, length 0
07:39:11.019576 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 54: (tos 0x40, ttl 64, id 1597, offset 0, flags 
[DF], proto TCP (6), length 40)
yyy.yyy.yyy.250.445 > 24.173.128.58.2762: Flags [R.], cksum 0x7500 (incorrect -> 0x0393), seq 0, ack 3247204676, win 0, length 0
07:39:45.378477 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 1605, offset 0, flags [DF], 
proto TCP (6), length 40)
zzz.zzz.zzz.27.1200 > 66.186.59.50.6667: Flags [R], cksum 0x1ad6 (incorrect -> 0x29e2), seq 269892722, win 0, length 0
07:49:35.036553 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 64, id 1706, offset 0, flags [DF], 
proto TCP (6), length 48)
zzz.zzz.zzz.27.22 > 66.135.60.231.9028: Flags [S.], cksum 0x1c60 (incorrect -> 0x8d8c), seq 2467225709, ack 1732566059, win 65535, options 
[mss 1460,sackOK,eol], length 0
07:49:38.035860 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 64, id 1708, offset 0, flags [DF], 
proto TCP (6), length 48)
zzz.zzz.zzz.27.22 > 66.135.60.231.9028: Flags [S.], cksum 0x1c60 (incorrect -> 0x8d8c), seq 2467225709, ack 1732566059, win 65535, options 
[mss 1460,sackOK,eol], length 0
07:49:44.035591 00:00:d1:ee:57:49 > 00:90:1a:a0:7b:bd, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 64, id 1712, offset 0, flags [DF], 
proto TCP (6), length 48)
zzz.zzz.zzz.27.22 > 66.135.60.231.9028: Flags [S.], cksum 0x1c60 (incorrect -> 0x8d8c), seq 2467225709, ack 1732566059, win 65535, options 
[mss 1460,sackOK,eol], length 0

You can see that packets which SHOULD be going out sf1 (yyy.yyy.yyy.250) are going out via sf0 (00:00:d1:ee:57:49), etc.

It would appear that not all packets exhibit this behavior, because I am able to get some traffic to pass through to machines on the inside.


Does anybody have any insight into what's going on, and possible remedies?
 
sf0 - WAN1
sf1 - WAN2
sf2 - WAN3
em0 - LAN

Is it correct?

try this:
Code: 
ipfw add 1010 prob 0.33 skipto 1060 ip from any to any in recv em0
ipfw add 1011 prob 0.5 skipto 1055 ip from any to any in recv em0

ipfw add 1040 setfib 0 ip from any to any via em0 keep-state
ipfw add 1050 allow ip from any to any via em0

ipfw add 1055 setfib 1 ip from any to any via em0 keep-state
ipfw add 1056 allow ip from any to any via em0

ipfw add 1060 setfib 2 ip from any to any via em0 keep-state
ipfw add 1070 allow ip from any to any via em0

ipfw add 1080 deny ip from any to 192.168.0.0/16 in recv sf[0-2]
ipfw add 1090 deny ip from 192.168.0.0/16 to any in recv sf[0-2]
ipfw add 10100 deny ip from any to 172.16.0.0/12 in recv sf[0-2]
ipfw add 10110 deny ip from 172.16.0.0/12 to any in recv sf[0-2]
ipfw add 10120 deny ip from any to 10.0.0.0/8 in recv sf[0-2]
ipfw add 10130 deny ip from 10.0.0.0/8 to any in recv sf[0-2]
ipfw add 10140 deny ip from any to 169.254.0.0/16 in recv sf[0-2]
ipfw add 10150 deny ip from 169.254.0.0/16 to any in recv sf[0-2]

ipfw nat 1 config log if sf0 same_ports reset deny_in
ipfw nat 2 config log if sf1 same_ports reset deny_in
ipfw nat 3 config log if sf2 same_ports reset deny_in

ipfw add 10170 nat 1 ip from any to any via sf0
ipfw add 10200 nat 2 ip from any to any via sf1
ipfw add 10210 nat 3 ip from any to any via sf2

ipfw add 10220 allow all from any to any

ipfw add 65534 deny all from any to any

set:
Code: 
sysctl net.inet.ip.fw.one_pass=0

you also should compile you kernel with
Code: 
options ROUTETABLES=2
and set default routes for "fib 1" and "fib 2".

You can use this rc.d script to do it (edit/customize it, or make anover one copy for fib 2):
Code: 
#!/bin/sh
# PROVIDE: SETFIB1
# REQUIRE: NETWORKING
# BEFORE: DAEMON
#
# Add the following lines to /etc/rc.conf to enable setfib -1 at startup
# setfib1 (bool): Set to "NO" by default.
#                Set it to "YES" to enable setfib1
# setfib1_defaultroute (str): Set to "" by default
#       Set it to ip address of default gateway for use in fib 1

. /etc/rc.subr

name="setfib1"
rcvar=`set_rcvar`

load_rc_config $name

[ -z "$setfib1_enable" ] && setfib1_enable="NO"
[ -z "$setfib1_defaultroute" ] && setfib1_defaultroute=""

start_cmd="${name}_start"
stop_cmd="${name}_stop"

setfib1_start()
{
if [ ${setfib1_defaultroute} ]
then
setfib 1 route delete default
setfib 1 route add default ${setfib1_defaultroute}
else
echo "Can not set default route for fib 1!"
fi
}

setfib1_stop()
{
setfib 1 route delete default
}
run_rc_command "$1"


This example should work.

---

In case of problems try to turn off TSO, RXCSUM and TXCSUM on sf[0-2] interfaces.
I know that RXCSUM/TXCSUM is a problem then you are using ipfw nat on em interfaces (may be on sf too?).
TSO is a well known limitation of libalias.
 
You must log in or register to reply here.
Back
Top