Currently using FreeBSD 7.2-Release with all ports up to date as my firewall. I've got a firewall/router with 3 NICs: 2x ADSL & 1 LAN. So far I've only been using one of the lines at a time, with the 2nd as a backup. Both ADSL lines have static IPs so I thought it's be handly to have the DNS record for our office to provide both IPs, this way even if one connection is down, the office can still be accessed remotely. My current pf.conf works fine, but for only a single connection, so I used an example pf.conf to configure a round-robin load balanced setup. Outbound traffic works perfectly, with the traffic using both ADSL lines. However I've had some issues with setting up the incoming traffic. In short, an incoming connection works fine if it's to the ADSL line that is the default gateway (tun1). If I attempt to access the office via the secondary ADSL line (tun0), I see the traffic coming in, but it is sent back out of the office on the primary ADSL line (tun1). How come traffic is being sent out the default gateway rather than the interface it came in on? What am I missing in (this very simple) ruleset? *The ISP for tun1 is actually routing the traffic back to the originator even though the source isn't correct, which in my opinion they shouldn't do. It works but obviously it can't be relied on. If however I swap the default gateway and rerun the test, the ISP tun0 doesn't route incorrectly sourced packets. ifconfig Code: tun0: flags=8151<UP,POINTOPOINT,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1492 inet xx.xx.115.90 --> xx.xx.112.1 netmask 0xffffffff Opened by PID 462 tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492 inet xx.xx.82.202 --> xx.xx.111.216 netmask 0xffffffff Opened by PID 470 netstat -rnf inet | grep default Code: default xx.xx.111.216 UGS 0 33943 tun1 Execute command from remote server (xx.xx.33.85) telnet xx.xx.82.202 22 Code: # tcpdump -ni tun0 port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type NULL (BSD loopback), capture size 96 bytes # tcpdump -ni tun1 port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun1, link-type NULL (BSD loopback), capture size 96 bytes 19:00:21.138403 IP xx.xx.33.85.2966 > xx.xx.82.202.22: S 1783025215:1783025215(0) win 65535 <mss 1380,nop,nop,sackOK> 19:00:21.138806 IP xx.xx.82.202.22 > xx.xx.33.85.2966: S 1046433002:1046433002(0) ack 1783025216 win 5840 <mss 1460,nop,nop,sackOK> 19:00:21.404552 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 1 win 65535 19:00:21.405412 IP xx.xx.82.202.22 > xx.xx.33.85.2966: P 1:26(25) ack 1 win 5840 19:00:21.728913 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 26 win 65510 19:00:24.253779 IP xx.xx.33.85.2966 > xx.xx.82.202.22: F 1:1(0) ack 26 win 65510 19:00:24.254186 IP xx.xx.82.202.22 > xx.xx.33.85.2966: F 26:26(0) ack 2 win 5840 19:00:24.424030 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 27 win 65510 Execute command from remote server (xx.xx.33.85) telnet xx.xx.115.90 22 Code: # tcpdump -ni tun0 port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type NULL (BSD loopback), capture size 96 bytes 19:01:08.721838 IP xx.xx.33.85.2967 > xx.xx.115.90.22: S 2245618778:2245618778(0) win 65535 <mss 1380,nop,nop,sackOK> 19:01:08.779974 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 3546884040 win 65535 19:01:09.052440 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 41 win 65495 19:01:10.676351 IP xx.xx.33.85.2967 > xx.xx.115.90.22: F 0:0(0) ack 41 win 65495 19:01:10.736404 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 42 win 65495 # tcpdump -ni tun1 port 22 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun1, link-type NULL (BSD loopback), capture size 96 bytes 19:01:08.722172 IP xx.xx.115.90.22 > xx.xx.33.85.2967: S 3546884039:3546884039(0) ack 2245618779 win 65535 <mss 1380,sackOK,eol> 19:01:08.820526 IP xx.xx.115.90.22 > xx.xx.33.85.2967: P 1:41(40) ack 1 win 65535 19:01:10.676488 IP xx.xx.115.90.22 > xx.xx.33.85.2967: . ack 2 win 65535 19:01:10.677654 IP xx.xx.115.90.22 > xx.xx.33.85.2967: F 41:41(0) ack 2 win 65535 pf.conf Take from http://www.openbsd.org/faq/pf/pools.html, but modified for my interfaces & additional rules for ssh Code: lan_net = "xx.xx.0.0/22" int_if = "rl0" ext_if1 = "tun0" ext_if2 = "tun1" ext_gw1 = "xx.xx.112.1" ext_gw2 = "xx.xx.111.216" # nat outgoing connections on each internet interface nat on $ext_if1 from $lan_net to any -> ($ext_if1) nat on $ext_if2 from $lan_net to any -> ($ext_if2) rdr on $ext_if1 proto tcp from any to $ext_if1 port ssh -> localhost rdr on $ext_if2 proto tcp from any to $ext_if2 port ssh -> localhost # default deny block in from any to any block out from any to any # pass all outgoing packets on internal interface pass out on $int_if from any to $lan_net # pass in quick any packets destined for the gateway itself pass in quick on $int_if from $lan_net to $int_if # load balance outgoing tcp traffic from internal network. pass in log on $int_if route-to \ { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \ proto tcp from $lan_net to any flags S/SA modulate state # load balance outgoing udp and icmp traffic from internal network pass in log on $int_if route-to \ { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \ proto { udp, icmp } from $lan_net to any keep state # general "pass out" rules for external interfaces pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state pass out on $ext_if1 proto { udp, icmp } from any to any keep state pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state pass out on $ext_if2 proto { udp, icmp } from any to any keep state pass in on $ext_if1 proto tcp from any to any port ssh pass in on $ext_if2 proto tcp from any to any port ssh # route packets from any IPs on $ext_if1 to $ext_gw1 and the same for # $ext_if2 and $ext_gw2 pass out log on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any pass out log on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any
You need reply-to statements (pf.conf(5)) for your 'pass in' rules to enforce symmetric routing on incoming connections. You'll probably also want an if-bound state policy. See http://kerneltrap.org/mailarchive/openbsd-misc/2007/7/14/152333 for an example. (not entirely sure about the if-bound/floating weirdness in there, but try if-bound first anyway)
Brilliant, it's a pity the OpenBSD pages didn't give an example of that. I've now got everything working perfectly as well as the ftp-proxy issue you helped me with earlier. Now I just have to have a good look through the man pages & docs to understand what exactly the if-bound policy does. For the moment, I've had to keep the gateway ips in tables, so that I can edit them once the rule-set has been loaded. This is due to my ISPs providing a different gateway whenever ppp has to redial. I'm currently using ppp.linkup to update the tables. This is the only method I found suggested when dealing with dynamic gateways. Thanks for all your help.
Stated simply: it will only create state for a connection on one interface at a time: on the interface the connection 'came in on'. This will prevent traffic from 'leaking out' of the wrong interface. The downside is that, instead of having the packet delivered (albeit using asymmetric routing using 'the wrong interface'), the packet won't be able to get out (the other interface doesn't have a state for that connection, so it refuses to pass the packet). The 'floating' policy allows states for connections on the other interfaces, so the packet will be allowed out on any interface. (switch in/out for traffic initiated locally)
Keep in mind that DNS usually works in a round-robin way. Meaning if one of your connections goes down you'll get connection problems 50% of the times you try to connect.
This is something I looked at. By taking down one of the connections and attempting to connect (using several browsers, .Net tcp client, ssh & telnet) and monitoring the traffic using wireshark, I found that the client would attempt to connect to one of the IPs, and if that timed out it would move on to the next IP returned by the DNS lookup. As you said, it works in a round-robin way. If one of the connections does go down, it will mean than connect times are greater as it has to wait for the attempt to timeout, but I've found that the client always manages to connect as long as at least one connection is up.
You could make a script and from time to time ping the gatway of both lines. If you have no reply from one gateway, pass all traffic to the other line.
I use pfSense (FreeBSD+PF combo) for failover and loadbalncing. It has a web-based front end. Few clicks and you are online. It takes care of everything.
I tried pfSense. It works with multiple WANs. But I cannot add php modules to run Drupal. I added Code:CUSTOM_PKG_LIST=........ and made many CD's. No pkg's are added into the CD's. Now I was trying OpenBSD. I even could not build its kernel. Now I am coming back to FreeBSD to see if possible to make freebsd FreeBSD to do pfSense and OpenBSD jobs.
