Multiple Gateways

mariourk

mariourk

Hi,

I have a bit of a luxury problem on my hands. Since last week we got antoher fiber connection to the internet. That's right. We have not one, but two fiber lines :cool:

However, this presents me with a problem I didn't anticipate. After activating the connection and testing it, everything wordked fine. So I hooked it up to our FreeBSD router/firewall. Pinging the modem works fine. I'm also able to access the webinterface. Browsing the internet works fine too. And a speedtest gives the expected results. So far so good. So I put in place a port forward, to simply forward all incoming traffic to FreeBSD and let PF handle it from there. Theoretically I should now be able to access FreeBSD from a remote location on the new fiber connection. I SSH-ed out to my server at home and tried to SSH back to our new IP-address. But no luck. I got nothing :(

I went over the PF-rules to see if somthing was configured wrong. But I couldn't find anything. So I hooked up a laptop directly to the new fiber line (directly means, directly to the modem). I changed the forward rule to the IP of the laptop and tried to SSH from a remote location again. It worked like a charm.

That meant that the problem was with my FreeBSD router/firewall. And I figured the problem was most likely in the PF-rules. So, I went over those again. But stll no luck. Until it dawned on me that all outgoing traffic is send over the old fiber connection, because that is the default gateway. So any incoming connection on the new fiber connection is never going to get an answer from the same IP-address. And that is most likely why it doesn't work.

The contract for the old fiber connection is still going for another two years. So I was hoping to gradually migrate everything to the new fiber connection, one service at a time. And only changing the affected DNS-records, one at a time. But right now, that doesn't seem possible. If I change the default gateway from the old to the new connection, I have to do it all. Probably during the night, or a weeking. Something I was hoping to avoid.

That got me thinking. Is it possible to configure multiple gateways? Can I tell FreeBSD to answer on the line where the packets came from in the first place? What's the best and most elegant way to handle this?

Kind regards,
Marinus ten Napel
 
mariourk said:
If I change the default gateway from the old to the new connection, I have to do it all. Probably during the night, or a weeking. Something I was hoping to avoid.
Click to expand...
setfib(1)() should be exactly what you are looking for.

A simple "use the interface where the request came in" is unfortunately not possible, FreeBSD routing goes like this:
a) Is the destination a local IP? If yes, send via loopback
b) Is the destination a directly connected network? If yes, send via the interface connected to that network
c) Everything else takes the default route
 
Try pf.conf(5) rules of this sort:
Code: 
table <martians> const { 0/8, 10/8, 127/8, 169.254/16, 172.16/12, 129.0.2/24, 192.168/16, 240/4 }

isp1_wan_if = igb1.2
isp1_wan_gw = 1.2.3.4

isp2_wan_if = igb1.29
isp2_wan_gw = 2.3.4.5

isp3_wan_if = igb1.4023
isp3_wan_gw = 3.4.5.6

# allowed inbound TCP and UDP services (inbound whitelist)
in_tcp_services = "{ ssh, www, https }"
in_udp_services = "{ ntp, openvpn }"
in_icmp_types = "{ echoreq, unreach }"

# enable external services and make sure output iface is same as input
pass in on $isp1_wan_if reply-to ( $isp1_wan_if $isp1_wan_gw ) inet proto icmp all icmp-type $in_icmp_types keep state
pass in on $isp2_wan_if reply-to ( $isp2_wan_if $isp2_wan_gw ) inet proto icmp all icmp-type $in_icmp_types keep state
pass in on $isp3_wan_if reply-to ( $isp3_wan_if $isp3_wan_gw ) inet proto icmp all icmp-type $in_icmp_types keep state

pass in on $isp1_wan_if reply-to ( $isp1_wan_if $isp1_wan_gw ) proto tcp from any to $isp1_wan_if port $in_tcp_services
pass in on $isp2_wan_if reply-to ( $isp2_wan_if $isp2_wan_gw ) proto tcp from any to $isp2_wan_if port $in_tcp_services
pass in on $isp3_wan_if reply-to ( $isp3_wan_if $isp3_wan_gw ) proto tcp from any to $isp3_wan_if port $in_tcp_services

pass in on $isp1_wan_if reply-to ( $isp1_wan_if $isp1_wan_gw ) proto udp from any to $isp1_wan_if port $in_udp_services
pass in on $isp2_wan_if reply-to ( $isp2_wan_if $isp2_wan_gw ) proto udp from any to $isp2_wan_if port $in_udp_services
pass in on $isp3_wan_if reply-to ( $isp3_wan_if $isp3_wan_gw ) proto udp from any to $isp3_wan_if port $in_udp_services

# multihomed2: redirect to respective gateway depending on outbound IP
pass out route-to ($isp1_wan_if $isp1_wan_gw) inet from $isp1_wan_if to ! <martians>
pass out route-to ($isp2_wan_if $isp2_wan_gw) inet from $isp2_wan_if to ! <martians>
pass out route-to ($isp3_wan_if $isp3_wan_gw) inet from $isp3_wan_if to ! <martians>
This should take care to send TCP/IP replies via respective connection. Not so with UDP, tho. The best I could is to run UDP services multiple times (once for each ISP, for example OpenVPN) and to bind them explicitly to the respective public interface. ntpd(8) automatically (and dynamically) does this. Give it a try, if it still doesn't work I'll see if there ain't anything else in my pf.conf(5).
 
First of all, apologies for the late response. I had to put this project on hold for a while.

But yesterday I finally had the time to pick it up again. I managed to get accessing the server over both lines working, thanks to the examples Bobi B. provided. I can now SSH to my server, using both IP-addresses. However, for some reason this doesn't work for forwarded ports. For example, I can access my webserver over IP-1, but not on IP-2.

I also cannot figure out how to route outgoing traffic over a preferred line. All the examples from the OpenBSD manual result is syntax errors.
 
I think the main problem is that this approach will always be somewhat of a hack, as such you can't really expect perfect results. Fact of the matter is that the whole TCP/IP routing concept simply doesn't provide for this. Falren called it FreeBSD routing but that's simply not true: this goes much deeper. Because you'll get the same results on pretty much any TCP/IP based environment. A general 'rule' is therefor not to use multiple nics for the same subnet because of the implied issues.

Note that I'm not saying it's fully impossible, others have already given some possible solutions, but I am saying that it will always be a little flakey.

(edit)

In the latest scenario though it would depend on how those ports were forwarded. If your Apache server listens on all interfaces and if both respond (try using telnet) then it is possible that this is caused by another routing hiccup.

If the forwarding gateway resides in the same subnet as both nics and it addresses IP 1 then how sure are you that your box isn't responding using IP 2? That could be a definite cause of problems.
 
mariourk said:
However, for some reason this doesn't work for forwarded ports. For example, I can access my webserver over IP-1, but not on IP-2.
Click to expand...
See if tag and tagged (pf.conf(5)) will help; myself never used forwarding, tho.

mariourk said:
I also cannot figure out how to route outgoing traffic over a preferred line.
Click to expand...
I used this naive approach:
Code: 
table <viaisp1> persist file "/etc/viaisp1.table"
table <viaisp2> persist file "/etc/viaisp2.table"
table <viaisp3> persist file "/etc/viaisp3.table"

# NAT respective LAN hosts to the public Internet (WAN)
nat inet from <viaisp1> to ! <martians> -> ($isp1_wan_if)
nat inet from <viaisp2> to ! <martians> -> ($isp2_wan_if)
nat inet from <viaisp3> to ! <martians> -> ($isp3_wan_if)
However that works for NATting different hosts via different lines. For locally hosted services see if setfib(1) will help. For the general case the heavy artillery is BGP.
 
I'm not sure where you got this, but I'm not using multiple nics for the same subnet. Or maybe I'm misunderstanding? My FreeBSD-server handles routing/firewalling for multiple subnets. And each subnet is connected to the FreeBSD-server on it's own nic. Two of these subnets are connected to our ISP's.

This is what is looks like:
  • igb0 -> Main LAN -> 1.1.1.0/23
  • igb1 -> ISP-1 -> 2.2.2.0/29 (this plugs directly into the server and gets an IP via DHCP from the ISP)
  • igb2 -> VoIP LAN -> 3.3.3.0/24
  • igb3 -> DMZ LAN -> 4.4.4.0/24
  • igb4 -> ISP-2 5.5.5.0/24 -> Modem -> external IP

Right now all incoming and outgoing traffic goes via igb1. Some ports are forwarded to an IP-address on either igb0, igb2, or igb3, depending on the service. Only SSH is running on the FreeBSD-server itself and is listening on all nics. And I managed to get a working SSH-connection when I login to to the external IP-address of either igb1, or igb4. All other (forwarded) services, not so much.

I implemented rules to forward ports from igb1 and igb4. But I cannot connect to any service when I'm trying to connect to the external IP-address in igb4. I suspect this is because the answer doesn't get routed back to the nic where it originally came from. But instead to the gateway on igb1. I did of cource forward the ports from the modem as well (otherwise SSH wouldn't work either, right? ;) )

What I want is to handle incoming connections, regardless whether they come from igb1, or igb4. This already works for SSH. I also want to route outgoing traffic to the gateway of choice. I'm not looking for a failover or load-balancing solution perse. I'm happy if I can send all outgoing traffic (destined for the internet) from igb0 to the gateway on igb4 and anything else to the gateway on igb1.
 
You must log in or register to reply here.
Back
Top