IPFW IPFW - Multi WAN

amr

amr

Hello,

Is it possible to have multi WAN with IPFW? I know it's possible with PF and PF does it well in pfSense. But I am running FreeBSD 13.1-RELEASE as my home firewall/gateway using some cool pkgs like net/kea, dns/unbound and FreeBSD's native firewall - IPFW with In-kernal NAT.
Here is an example code for one WAN/Gateway and it's running without any issues:
Code: 
ipfw -q -f flush

add="ipfw -q add"
wan="tun0"
lan="em1"

ipfw -q nat 1 config if $wan reset same_ports unreg_only

# Allow all lo0 traffic:
$add 100 pass all from any to any via $lan
$add 200 pass all from any to any via lo0

# Reject spoofing:
$add 300 deny ip from any to any not antispoof in

# NAT rule for incoming packets.
# NAT in rule must appear BEFORE check-state.
$add 400 nat 1 ip from any to any via $wan in

$add 500 check-state

# Port redirection rules to internal, skipto 10000
# ...

# Prevent lan from spamming
$add 1000 deny ip from not me to any smtp,smtps via $wan out

# Allow all other outgoing connections
$add 2000 skipto 10000 tcp from any to any via $wan out keep-state
$add 2100 skipto 10000 udp from any to any via $wan out keep-state

# I server web pages too:
$add 5000 allow tcp from any to me http,https via $wan in keep-state

$add 9000 allow icmp from any to any

$add 9999 deny all from any to any

# NAT rule for outgoing packets
$add 10000 nat 1 ip from any to any via $wan out

Update:
My two WANs from different ISPs, and I connect to each ISP using PPPoE with tun0 and tun1 interfaces
So besides ipfw configuration I need to implement routing tables to group my two gateways and to be honest I don't know how to make this job done plus I don't know the right tools for it.

Thanks,
 
Yes it's possible to have multiple WAN interfaces in IPFW. It depend what you want to achieve and what your ISP provide to you.

The best option is to have your own AS with IPv4 block so you can advertise it over BGP to other AS (Multihomed Environment). net/quagga
Other option is to be connected to the same ISP which provide you an IPv4 block and you advertise this IPv4 address over two lines (Primary/Backup) it's also called dual-homed to single ISP topology. My current network setup is like this but with Cisco routers.
And the cheapest option is to have independent ISP providing you with no public IPv4 address without any dynamic routing then you will need to track(monitor using ping) the availability of the line using static route thought one of the ISP and when the line is down to change the default gateway to the other line. This is also known as SD-WAN.
If you want to have some kind of rr-load balance then look for LSNAT option examples from userland natd(8) or in-kernel nat in ipfw(8)

Good example of many different topologies can be found here
 
What I want to achieve is to use both ISPs connections to load balance the traffic for my LANs
 
Hello,

At the moment, the multi-WAN is working using FIB.

My first ISP using setfib 0 - the default
Code: 
root@nouh:/home/amr # setfib 0 ping -c 5 google.com
PING google.com (172.217.19.46): 56 data bytes
64 bytes from 172.217.19.46: icmp_seq=0 ttl=112 time=48.515 ms
64 bytes from 172.217.19.46: icmp_seq=1 ttl=112 time=48.274 ms
64 bytes from 172.217.19.46: icmp_seq=2 ttl=112 time=47.977 ms
64 bytes from 172.217.19.46: icmp_seq=3 ttl=112 time=48.467 ms
64 bytes from 172.217.19.46: icmp_seq=4 ttl=112 time=48.501 ms

--- google.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 47.977/48.347/48.515/0.204 ms

My second ISP using setfib 1
Code: 
root@nouh:/home/amr # setfib 1 ping -c 5 google.com
PING google.com (172.217.19.46): 56 data bytes
64 bytes from 172.217.19.46: icmp_seq=0 ttl=118 time=42.322 ms
64 bytes from 172.217.19.46: icmp_seq=1 ttl=118 time=42.957 ms
64 bytes from 172.217.19.46: icmp_seq=2 ttl=118 time=42.314 ms
64 bytes from 172.217.19.46: icmp_seq=3 ttl=118 time=42.508 ms
64 bytes from 172.217.19.46: icmp_seq=4 ttl=118 time=42.035 ms

--- google.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 42.035/42.427/42.957/0.305 ms

So please can you guide me how to use the FIBs with IPFW to load balance the traffic.
 
Thanks for your advice,
I've read it.
Code: 
MULTIPLE INSTANCES
     It    is not so uncommon to have a need of aliasing to several external IP
     addresses.     While this traditionally was achieved by running several natd
     processes with independent    configurations,    natd can have multiple alias-
     ing instances in a    single process,    also allowing them to be not so    inde-
     pendent of    each other.  For example, let us see a common task of load
     balancing two channels to different providers on a    machine    with two ex-
     ternal interfaces `sis0' (with IP 1.2.3.4)    and `sis2' (with IP 2.3.4.5):

             net 1.2.3.0/24
       1.2.3.1 ------------------ sis0
       (router)           (1.2.3.4)
                            net    10.0.0.0/24
                         sis1 ------------------- 10.0.0.2
                      (10.0.0.1)
             net 2.3.4.0/24
       2.3.4.1 ------------------ sis2
       (router)           (2.3.4.5)

     Default route is out via `sis0'.

     Interior machine (10.0.0.2) is accessible on TCP port 122 through both
     exterior IPs, and outgoing    connections choose a path randomly between
     `sis0' and    `sis2'.

     The way this works    is that    natd.conf builds two instances of the aliasing
     engine.

     In    addition to these instances' private divert(4) sockets,    a third    socket
     called the    "globalport" is    created; packets sent to natd via this one
     will be matched against all instances and translated if an    existing entry
     is    found, and unchanged if    no entry is found.  The    following lines    are
     placed into /etc/natd.conf:

       log
       deny_incoming
       verbose

       instance default
       interface sis0
       port    1000
       redirect_port tcp 10.0.0.2:122 122

       instance sis2
       interface sis2
       port    2000
       redirect_port tcp 10.0.0.2:122 122

       globalport 3000

     And the following ipfw(8) rules are used:

       ipfw    -f flush

       ipfw    add     allow ip from any to any via sis1

       ipfw    add     skipto    1000 ip    from any to any    in via sis0
       ipfw    add     skipto    2000 ip    from any to any    out via    sis0
       ipfw    add     skipto    3000 ip    from any to any    in via sis2
       ipfw    add     skipto    4000 ip    from any to any    out via    sis2

       ipfw    add 1000 count ip from any to any

       ipfw    add     divert    1000 ip    from any to any
       ipfw    add     allow ip from any to any

       ipfw    add 2000 count ip from any to any

       ipfw    add     divert    3000 ip    from any to any

       ipfw    add     allow ip from 1.2.3.4 to any
       ipfw    add     skipto    5000 ip    from 2.3.4.5 to    any

       ipfw    add     prob .5 skipto    4000 ip    from any to any

       ipfw    add     divert    1000 ip    from any to any
       ipfw    add     allow ip from any to any

       ipfw    add 3000 count ip from any to any

       ipfw    add     divert    2000 ip    from any to any
       ipfw    add     allow ip from any to any

       ipfw    add 4000 count ip from any to any

       ipfw    add     divert    2000 ip    from any to any

       ipfw    add 5000 fwd 2.3.4.1 ip    from 2.3.4.5 to    not 2.3.4.0/24
       ipfw    add     allow ip from any to any

     Here the packet from internal network to Internet goes out    via `sis0'
     (rule number 2000)    and gets caught    by the globalport socket (3000).  Af-
     ter that, either a    match is found in a translation    table of one of    the
     two instances, or the packet is passed to one of the two other divert(4)
     ports (1000 or 2000), with    equal probability.  This ensures that load
     balancing is done on a per-flow basis (i.e., packets from a single    TCP
     connection    always flow through the    same interface).  Translated packets
     with source IP of a non-default interface (`sis2')    are forwarded to the
     appropriate router    on that    interface.

Since I am using Inkernal-NAT the syntax is different, and actually I don't know how to make it work and in the same time the man page natd() didn't mentioned anything about FIBs.
So please if you could write an example of FIBs with Multi NAT instance of Inkernal-NAT will be appreciated so much.
 
You still need some monitoring script to detect when some of the wan interface is down.

#!/bin/sh

# Flush out ipfw rules
ipfw -q -f flush

# Set rules command prefix
fwcmd="ipfw -q add"

WAN0="em0"
WAN1="em1"
LAN="em2"

# NAT config
ipfw nat 1 config if ${WAN0} deny_in reset same_ports
ipfw nat 2 config if ${WAN1} deny_in reset same_ports

# dynamic states
$fwcmd 10 check-state :wan0
$fwcmd 20 check-state :wan1

$fwcmd 50 allow tcp from any to me dst-port 22

# 50% round robin across the wan0 and wan1 from lan
$fwcmd 100 prob .5 skipto 300 ip from any to any in recv $LAN

# skip to the nat section depending of traffic recv/xmit
$fwcmd 110 skipto 200 ip from any to any in recv $WAN0
$fwcmd 120 skipto 200 ip from any to any out xmit $WAN0
$fwcmd 130 skipto 300 ip from any to any in recv $WAN1
$fwcmd 140 skipto 300 ip from any to any out xmit $WAN1

# NAT WAN0
$fwcmd 200 nat 1 ip from any to any in recv $WAN0
$fwcmd 201 skipto 210 ip from any to any keep-state :wan0
$fwcmd 210 setfib 0 ip from any to any in recv $LAN
$fwcmd 220 nat 1 ip from any to any out xmit $WAN0
$fwcmd 299 skipto 65000 ip from any to any

# NAT WAN1
$fwcmd 300 nat 2 ip from any to any in recv $WAN1
$fwcmd 301 skipto 310 ip from any to any keep-state :wan1
$fwcmd 310 setfib 1 ip from any to any in recv $LAN
$fwcmd 320 nat 2 ip from any to any out xmit $WAN1

$fwcmd 65000 allow ip from any to any
Click to expand...

Edit:
rule 10 check-state :wan1 :wan0
rule 20 check-state :wan2 :wan1
 
Last edited:
First I would like to thank VladiBG for his help and support, it actually works
But I am facing problem, which is all traffic are open
So I've tried so many syntax to block all the traffic and open only what I wants but I couldn't make it work because I used to use via not recv or xmit
 
I want to block all traffic and allow outbound connection from lan to outside for ports http,https and icmp
And allow udp connection to following dns to restrict my network from using other public dns
  • 208.67.222.123
  • 208.67.220.123
And if I am using port_redirect 192.168.1.1:80 80
How to write a rule for it.
 
I don't have 2 ISP at home to test it but it should look like this:

ipfw delete 65000
ipfw add 65005 allow icmp from any to any
ipfw add 65010 allow udp from any to 208.67.222.123 53 keep-state
ipfw add 65015 allow udp from any to 208.67.220.123 53 keep-state
ipfw add 65020 allow tcp from any to any 80,443 setup keep-state
Click to expand...

Or if you want to deny your ip to respond to ping then instead of allowing icmp from any to any use
ipfw add 65005 allow icmp from me to any keep-state
ipfw add 65006 allow icmp from 192.168.1.0/24 to any keep-state
Click to expand...
 
Hello,
the following code using the default fib 0 how I can make it use fib 1 for example
Code: 
#!/bin/sh

ipfw -q -f flush        # Flush out the list

# Rules command prefix
ks="keep-state"
skip="skipto 1000"
amr="ipfw -q add"
wan="tun0"              # WAN interface
lan="em0"               # LAN interface
dns="208.67.222.123,208.67.220.123"
good="192.168.100.0/24"

ipfw disable one_pass
ipfw -q nat 1 config if $wan deny_in same_ports unreg_only reset

# Allow all traffic on LAN and on the loopback interface
$amr 0100 allow all from any to any via $lan
$amr 0150 allow all from any to any via lo0

# The reassemble rule
$amr 0200 reass all from any to any in

# In-kernal NAT
$amr 0250 nat 1 ip from any to any in via $wan

# Allow the packet through if it matches an existing entry in the dynamic rules table
$amr 0300 check-state

# Allow access to OpenDNS only
$amr 0350 $skip udp from any to $dns domain out via $wan $ks

# Allowed ports
$amr 0450 $skip ip from $good to any out via $wan $ks

# Allow ping
$amr 0600 $skip icmp from any to any out via $wan $ks

# Deny and log everything else
$amr 0650 deny log all from any to any

# Skipto location for outbound stateful rules
$amr 1000 nat 1 ip from any to any out via $wan
$amr 1050 allow ip from any to any
 
VladiBG said:
Did you use my ipfw rules from post #7 ? What is the output of ipfw show?
Click to expand...
Yes, I've used it

Code: 
oot@nouh:/home/amr # ipfw show
00005  3690   875078 queue 1 ip from any to any xmit tun0 out
00010  8007  4174013 queue 2 ip from any to any recv tun0 in
00015 15129  2725740 queue 1 ip from any to any xmit tun1 out
00020 20917 17793306 queue 2 ip from any to any recv tun1 in
00025    48     3572 allow ip from any to any via lo0
00030 47428 25551843 allow ip from any to any via em0
00045     0        0 check-state :wan0
00050     0        0 check-state :wan1
00051     0        0 allow tcp from any to me 22
00055     0        0 prob 0.500000 skipto 300 ip from any to any in recv em0
00110     0        0 skipto 200 ip from any to any in recv tun0
00120     0        0 skipto 200 ip from any to any out xmit tun0
00130     0        0 skipto 300 ip from any to any in recv tun1
00140     0        0 skipto 300 ip from any to any out xmit tun1
00200     0        0 nat 1 ip from any to any in recv tun0
00201     0        0 skipto 210 ip from any to any keep-state :wan0
00210     0        0 setfib 0 ip from any to any in recv em0
00220     0        0 nat 1 ip from any to any out xmit tun0
00299     0        0 skipto 65000 ip from any to any
00300     0        0 nat 2 ip from any to any in recv tun1
00301     0        0 skipto 310 ip from any to any keep-state :wan1
00310     0        0 setfib 1 ip from any to any in recv em0
00320     0        0 nat 2 ip from any to any out xmit tun1
65005     0        0 allow icmp from any to any
65010     0        0 allow udp from any to 208.67.222.123 53 keep-state :default
65015     0        0 allow udp from any to 208.67.220.123 53 keep-state :default
65020     0        0 allow tcp from any to any 80,443 setup keep-state :default
65030     0        0 allow tcp from any to me 22
65535  1754   132851 deny ip from any to any

I've locked myself so I've added

Code: 
00025 allow ip from any to any via lo0
00030 allow ip from any to any via em0
 
You need to allow outgoing traffic from you after the check-state instead of catching all traffic at your rule 30.
It should look like this where 192.168.1.0/24 is your LAN

00010 check-state :wan0
00020 check-state :wan1
00030 allow tcp from any to me 22
00035 allow tcp from me to any established
00040 allow tcp from me to any setup keep-state
00050 allow udp from me to any keep-state
00060 allow icmp from me to any keep-state
00070 allow ipv6-icmp from me to any keep-state
00100 prob 0.500000 skipto 300 ip from any to any in recv em0
00110 skipto 200 ip from any to any in recv tun0
00120 skipto 200 ip from any to any out xmit tun0
00130 skipto 300 ip from any to any in recv tun1
00140 skipto 300 ip from any to any out xmit tun1
00200 nat 1 ip from any to any in recv tun0
00201 skipto 210 ip from any to any keep-state :wan0
00210 setfib 0 ip from any to any in recv em0
00220 nat 1 ip from any to any out xmit tun0
00299 skipto 65000 log ip from any to any
00300 nat 2 ip from any to any in recv tun1
00301 skipto 310 ip from any to any keep-state :wan1
00310 setfib 1 ip from any to any in recv em0
00320 nat 2 ip from any to any out xmit tun1
65000 allow icmp from 192.168.1.0/24 to any keep-state :default
65010 allow udp from 192.168.1.0/24 to 208.67.222.123 53 keep-state
65020 allow tcp from 192.168.1.0/24 to any 80,443 setup keep-state
Click to expand...
 
there is no internet access for LAN I don't know why
Code: 
00005   17  13087 queue 1 ip from any to any xmit tun0 out
00010  384  31507 queue 2 ip from any to any recv tun0 in
00015   87   8359 queue 1 ip from any to any xmit tun1 out
00020   87  10205 queue 2 ip from any to any recv tun1 in
00025    0      0 check-state :wan0
00030    0      0 check-state :wan1
00035  126  10882 allow tcp from any to me 22
00040  125  19158 allow tcp from me to any established
00045    0      0 allow tcp from me to any setup keep-state :default
00050   20   1634 allow udp from me to any keep-state :default
00055    0      0 allow icmp from me to any keep-state :default
00060   79  10213 prob 0.500000 skipto 300 ip from any to any in recv em0
00110    0      0 skipto 200 ip from any to any in recv tun0
00120    0      0 skipto 200 ip from any to any out xmit tun0
00130    0      0 skipto 300 ip from any to any in recv tun1
00140    0      0 skipto 300 ip from any to any out xmit tun1
00200    0      0 nat 1 ip from any to any in recv tun0
00201  486  55462 skipto 210 ip from any to any keep-state :wan0
00210  118  26162 setfib 0 ip from any to any in recv em0
00220    0      0 nat 1 ip from any to any out xmit tun0
00299  486  55462 skipto 65000 ip from any to any
00300    0      0 nat 2 ip from any to any in recv tun1
00301  140  21972 skipto 310 ip from any to any keep-state :wan1
00310  130  17808 setfib 1 ip from any to any in recv em0
00320    0      0 nat 2 ip from any to any out xmit tun1
65000    0      0 allow icmp from 192.168.1.0/24 to any keep-state :default
65010    0      0 allow udp from 192.168.1.0/24 to 208.67.222.123,208.67.220.123 53 keep-state :default
65020   26   9167 allow tcp from 192.168.1.0/24 to any 80,443 setup keep-state :default
65535 1468 129078 deny ip from any to any
 
I've removed my queue and still there is no internet access to LAN
Code: 
root@nouh:/home/amr # ipfw show

00025      0         0 check-state :wan0
00030      0         0 check-state :wan1
00035    219     17482 allow tcp from any to me 22
00040    152     19114 allow tcp from me to any established
00045      0         0 allow tcp from me to any setup keep-state :default
00050    119     14891 allow udp from me to any keep-state :default
00055     80      6612 allow icmp from me to any keep-state :default
00060   1580    154740 prob 0.500000 skipto 300 ip from any to any in recv em0
00110    637    104879 skipto 200 ip from any to any in recv tun0
00120      0         0 skipto 200 ip from any to any out xmit tun0
00130 297875 438509681 skipto 300 ip from any to any in recv tun1
00140      0         0 skipto 300 ip from any to any out xmit tun1
00200    637    104879 nat 1 ip from any to any in recv tun0
00201   7853    959518 skipto 210 ip from any to any keep-state :wan0
00210   5228    565149 setfib 0 ip from any to any in recv em0
00220    584     80031 nat 1 ip from any to any out xmit tun0
00299   7269    879487 skipto 65000 ip from any to any
00300 297875 438509681 nat 2 ip from any to any in recv tun1
00301 598987 451550249 skipto 310 ip from any to any keep-state :wan1
00310 152734   6755273 setfib 1 ip from any to any in recv em0
00320 148558   6343488 nat 2 ip from any to any out xmit tun1
65000   2259    153945 allow icmp from 192.168.1.0/24 to any keep-state :default
65010      0         0 allow udp from 192.168.1.0/24 to 208.67.222.123,208.67.220.123 53 keep-state
 :default
65020 447206 445099614 allow tcp from 192.168.1.0/24 to any 80,443 setup keep-state :default
65535 8242 833459 deny ip from any to any
 
VladiBG said:
When the packet exits via your queue rule it's not passed back into the firewall.
Click to expand...
So one_pass would need to be disabled. Is that the case here?

Also, can we be reminded of current entire configuration including nat config statements etc. Trying to follow ...
 
Then change your rules from 65000-65020 replace 192.168.1.0/24 with your LAN subnet 192.168.100.0/24.
Also there's some device with ip address from 192.168.1.0/24 which is trying to access port 80,443 so it's better to check which one is it.

Do you know how to trace the packet flow through your firewall?

My advice is to use a firewall which you understand how it's work and how the packets are filtered.
I personally find IPFW to be more hard to trace especially with in-kernel libalias as there's no implementation to show the current aliasing links.


smithi said:
So one_pass would need to be disabled. Is that the case here?
Click to expand...
Yes, if you need to process the packet after it came back from the queue (nat, forwarding etc.)
 
VladiBG said:
Then change your rules from 65000-65020 replace 192.168.1.0/24 with your LAN subnet 192.168.100.0/24.
Also there's some device with ip address from 192.168.1.0/24 which is trying to access port 80,443 so it's better to check which one is it.
Click to expand...
I am using subnet 192.168.100.0/24 and for thread sake I used 192.168.1.0/24, so even when I used 192.168.100.0/24 there is no internet access for the LAN, I think it because we are blocking traffic for the lo0, since I am using dns/unbound as a resolver.


VladiBG said:
Do you know how to trace the packet flow through your firewall?
Click to expand...
Unfortunately no.

VladiBG said:
My advice is to use a firewall which you understand how it's work and how the packets are filtered.
I personally find IPFW to be more hard to trace especially with in-kernel libalias as there's no implementation to show the current aliasing links.
Click to expand...
I really understand IPFW with Inkernal-NAT with state-full rules for one instance only as of FreeBSD Handbook, And I really appreciate your valuable time because I am learning from you, My only problem with the multi-instances inkernal-NAT which you teach me that all traffic are open. And I didn't find any implementation like this on google, bing and duckduckgo.
 
You must log in or register to reply here.
Back
Top