IPFW IPFW - Multi WAN

[VladiBG]

Here is example how you can troubleshoot your config using firewall_logging="yes" in /etc/rc.conf and observing /var/log/security

Note: libalias(3) doesn't support showing the current aliasing table for the NAT.

Network

LAN em0: 192.168.1.0/24
ISP1 tun0: 10.0.100.0/24
ISP2 tun1: 10.0.200.0/24

You need your em0 interface to be reachable in both fib0 and fib1

# sysctl net.add_addr_allfibs
net.add_addr_allfibs: 1
# netstat -rn4
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 10.0.100.1 UGS tun0
127.0.0.1 link#1 UH lo0
192.168.1.0/24 link#3 U em0
10.0.100.1 link#3 UHS lo0
10.0.200.0/24 link#4 U tun1
10.0.100.0/24 link#2 U tun0
10.0.100.115 link#2 UHS lo0
root@r1:/etc # setfib 1 netstat -rn4
Routing tables (fib: 1)

Internet:
Destination Gateway Flags Netif Expire
default 10.0.200.1 UGS tun1
127.0.0.1 link#1 UH lo0
192.168.1.0/24 link#3 U em0
10.0.200.0/24 link#4 U tun1
10.0.200.2 link#4 UHS lo0
10.0.100.0/24 link#2 U tun0

# ipfw show
00010 0 0 check-state :wan0
00020 0 0 check-state :wan1
00030 0 0 allow tcp from any to me 22
00040 0 0 allow tcp from me to any established
00050 0 0 allow tcp from me to any setup keep-state :default
00060 0 0 allow udp from me to any keep-state :default
00070 0 0 allow icmp from me to any keep-state :default
00100 1 84 prob 0.500000 skipto 300 log ip from any to any in recv em0
00110 4 788 skipto 200 log ip from any to any in recv tun0
00120 0 0 skipto 200 log ip from any to any out xmit tun0
00130 1 84 skipto 300 log ip from any to any in recv tun1
00140 0 0 skipto 300 log ip from any to any out xmit tun1
00200 4 788 nat 1 log ip from any to any in recv tun0
00201 0 0 skipto 210 log ip from any to any keep-state :wan0
00210 0 0 setfib 0 log ip from any to any in recv em0
00220 0 0 nat 1 log ip from any to any out xmit tun0
00299 0 0 skipto 65000 log ip from any to any
00300 1 84 nat 2 log ip from any to any in recv tun1
00301 3 252 skipto 310 log ip from any to any keep-state :wan1
00310 1 84 setfib 1 log ip from any to any in recv em0
00320 1 84 nat 2 log ip from any to any out xmit tun1
65000 2 168 allow log icmp from 192.168.1.0/24 to any keep-state :default
65010 0 0 allow log udp from 192.168.1.0/24 to 8.8.8.8 53 keep-state :default
65020 0 0 allow log tcp from 192.168.1.0/24 to any 80,443 setup keep-state :default
65535 0 0 deny ip from any to any

Packet trace via ICMP echo request from client inside lan with ip 192.168.1.2 -> gw:192.168.1.1 -> 10.0.100.1 or 10.0.200.1 -> 8.8.8.8

src:192.168.1.2 dst:8.8.8.8

ip_input()

Jul 11 19:35:17 r1 kernel: ipfw: 100 SkipTo 300 ICMP:8.0 192.168.1.2 8.8.8.8 in via em0
Jul 11 19:35:17 r1 kernel: ipfw: 301 SkipTo 310 ICMP:8.0 192.168.1.2 8.8.8.8 in via em0

dynamic state created

301 (skipto) STATE icmp 192.168.1.2 0 <-> 8.8.8.8 0 :wan1

Jul 11 19:35:17 r1 kernel: ipfw: 310 SetFib 1 ICMP:8.0 192.168.1.2 8.8.8.8 in via em0
Jul 11 19:35:17 r1 kernel: ipfw: 65000 Accept ICMP:8.0 192.168.1.2 8.8.8.8 in via em0

dynamic state created

65000 (allow) STATE icmp 192.168.1.2 0 <-> 8.8.8.8 0 :default

ip_forward()
ip_output()
Dynamic state match

301 (skipto) STATE icmp 192.168.1.2 0 <-> 8.8.8.8 0 :wan1

Jul 11 19:35:17 r1 kernel: ipfw: 301 SkipTo 310 ICMP:8.0 192.168.1.2 8.8.8.8 out via tun1
Jul 11 19:35:17 r1 kernel: ipfw: 320 Nat ICMP:8.0 192.168.1.2 8.8.8.8 out via tun1

Nat aliasing created
libalias()
(192.168.1.2, 8, 8.8.8.8, 0, 0, 0, icmp)

----
echo reply
src:8.8.8.8 dst: 10.0.200.2
ip_input()

Jul 11 19:35:17 r1 kernel: ipfw: 130 SkipTo 300 ICMP:0.0 8.8.8.8 10.0.200.2 in via tun1
Jul 11 19:35:17 r1 kernel: ipfw: 300 Nat ICMP:0.0 8.8.8.8 10.0.200.2 in via tun1

NAT de-aliasing match
libalias()
(192.168.1.2, 8, 8.8.8.8, 0, 0, 0, icmp)

ip_forward()
ip_output()
Dynamic state match

301 skipto STATE icmp 192.168.1.2 0 <-> 8.8.8.8 0 :wan1

Jul 11 19:35:17 r1 kernel: ipfw: 301 SkipTo 310 ICMP:0.0 8.8.8.8 192.168.1.2 out via em0

Dynamic state match

65000 allow STATE icmp 192.168.1.2 0 <-> 8.8.8.8 0 :default

Jul 11 19:35:17 r1 kernel: ipfw: 65000 Accept ICMP:0.0 8.8.8.8 192.168.1.2 out via em0