Multiple default gateways

Colleagues, please tell me how to create routing for my case and whether it is possible.

I have a host, and it has several interfaces accessible from different provider networks. For example, ng0 and ue0.
I need to make it so that if a connection is established by a remote client through the ng0 interface, then the response packets should go to it. And if through ue0 - then they should go to ue0.
This is not an intranet, but a full-fledged Internet. How to build routing correctly?

I am grateful for the advice,
Ogogon.
 
There can only be one (1) default gateway[*]. That is the nature of a default. Cisco typically calls it "gateway of last resort", which is honestly a much better description. It's where things are routed to if no other routing entry applies.

[*] per routing table. Each routing table can have it's own default, but there can be only one for each table.
 
you can use ipfw fwd to the respective gw on outgoing packets if their src matches the local addr you have from respective provider
that i used in the past and it worked
or probably ipfw setfib will work too
 
you can use ipfw fwd to the respective gw on outgoing packets if their src matches the local addr you have from respective provider
that i used in the past and it worked
or probably ipfw setfib will work too
Thank you. I am much more interested in aspects that I have not found described anywhere.

For example. Let's assume that the connection was established via the ue0 interface, affiliated with the FIB2 routing table. How can I make sure that the response packets of this connection are also sent via the routing described in FIB2, and not FIB0? Is there such a possibility?
 
Colleagues, please tell me how to create routing for my case and whether it is possible.

I have a host, and it has several interfaces accessible from different provider networks. For example, ng0 and ue0.
I need to make it so that if a connection is established by a remote client through the ng0 interface, then the response packets should go to it. And if through ue0 - then they should go to ue0.
This is not an intranet, but a full-fledged Internet. How to build routing correctly?

I am grateful for the advice,
Ogogon.

May I ask what service(s) you need to provide on both your public interfaces? (I'm asking because if UDP is involved, that complicates things / effects my recommendations how to solve this)
 
Thank you. I am much more interested in aspects that I have not found described anywhere.

For example. Let's assume that the connection was established via the ue0 interface, affiliated with the FIB2 routing table. How can I make sure that the response packets of this connection are also sent via the routing described in FIB2, and not FIB0? Is there such a possibility?

Once upon a time, when using the "FIB approach" to make a FreeBSD host "multi-homed", all one needed to do was:
  • Set net.fibs=N in /boot/loader.conf (where N is the number of desired routing tables; e.g.: 4).
  • Add a route add default 5.6.7.8 -fib 2 somewhere in the system startup (assuming that 5.6.7.8 is the gateway for your "secondary" Internet connection.
  • Start a separate daemon for each service you want to offer to Internet clients -- one bound to your primary public IP, the other bound to your secondary public IP, and the daemon on your secondary IP started with with setfib n; e.g.: setfib 2 /usr/local/etc/rc.d/yourdaemon.
One nuance here is that every process has a FIB# associated with it (can be viewed via sysctl net.my_fibnum and set via setfib n). And that FIB is inherited by any spawned-children.

But since Nov 8 2020 (src commit 2d39824195933c173bbfc9b31773070202d2e30e), the default for sysctl net.add_addr_allfibs changed from 1 to 0, which means that non-zero FIBs no-longer automatically get any routes for any of your local interfaces (other than lo0), so if you are using FIBs in the "modern world", you apparently need to either reset sysctl net.add_addr_allfibs=1 before network startup (to revert to the "old" behavior), -OR- manually add interface routes for all of your relevant non-zero FIBs before you will be able to add a default route in one of your non-zero FIBs to your secondary gateway.
Here's an example how to add an interface route to a non-zero FIB:
route add -net 192.168.0.0/24 -iface vlan0 -fib 2
When net.add_addr_allfibs=0, at least one local interface route must be configured in that FIB before a default route could be added to it.

If you're not certain your secondary FIB default-route is working properly, I recommend verifying it; e.g.:
setfib 2 traceroute 8.8.8.8
then see if the trace shows your packets going through your alternate gateway.

If you're running one or more services that were architected to do only one "wildcard" listen/bind and you expect only that one process to respond from all your public IPs, then the setfib approach probably won't work cleanly, because the kernel (by default) will send all outbound packets through the default gateway associated with that process's current FIB. Like SirDice said, there can be "only one" default route for any given process.

An implication if you don't configure this correctly would be that a packet with provider X's source-address might be sent out through provider Y (with your provider X source-address), but your provider Y won't get bidirectional packet-flow when provider X is down, probably circumvents the reason you are doing multi-homing. And some providers might blackhole packets being sent from their network if you put a source-address on an outbound packet that is not within their network (i.e.: they might filter packets that seem "spoofed"). So configuring this correctly is required if you want it to work as-intended.

Bottom line: if you can run separate daemons (each bound to each of your public IPs), I think that's likely to work cleanly. But if you want only one daemon process to work with a wildcard-bind, then covacat's suggestion to use ipfw fwd rules probably is the best approach. FWIW, I have used ipfw fwd for this myself, it should work for TCP services, but if you have any UDP involved, then your daemon source code likely will need to be changed to open/bind one UDP socket per public IP, otherwise the daemon will not be able to select the desired/proper source address on outbound UDP packets.

To be clear, you probably wouldn't need to configure/use any FIBs if you use ipfw fwd, but like I said, that probably won't work cleanly if any UDP wildcard-binds are involved.

Let me know if I may be of any assistance with configuring ipfw for this..
 
Hello! well I've managed to setup somewhat alike on couple of hosts. First, you'd as it was already pointed out above setup separate fib for each channel. In my case I've two internet service providers, and in the following example assume that one ISP connected to em0, the 2nd one to re1. At /boot/loader.conf there is
net.fibs=2
net.add_addr_allfibs=1


and routing is set up somewhat like that:
$ netstat -rnF0
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default ISP1_gw UGS em0
...

$ netstat -rnF1
Routing tables (fib: 1)

Internet:
Destination Gateway Flags Netif Expire
default ISP2_gw UGS re1
...


In ipfw rules, at the beginning I've these rules for that:
01100 reass ip from any to me in
01200 count tag 1 ip from any to me in via re1
01300 fwd ISP2_gw log logamount 1000 ip from ISP2_ip to any out via em0
01400 fwd ISP1_gw log logamount 1000 ip from ISP1_ip to any out via re1


then goes rules but not check or set state, and finally NAT part:

03600 nat global ip from MY_LANS to not me out { xmit em0 or xmit re1 }
03700 nat 3 ip from MY_LANS to not me out xmit em0
03800 nat 3 ip from any to me in recv em0
03900 nat 7 ip from MY_LANS to not me out xmit re1
04000 nat 7 ip from any to me in recv re1
04200 setfib 1 ip from any to any tagged 1 keep-state :otherisp
04300 check-state :default


this setup allows me to have separate ssh / wireguard connection on specific ISP line (that can be easy controlled by tcpdump); I also can, for example, test any link speed by setfib 1 speedtest --secure (or setfib 0), etc.

As far as I remember this setup is my final version, I 'v tried several variants and this is satisfactory. It works quite as expected (at least I hope so =) for me and if this is of any help to your task I'd quite delighted about this fact
 
Do the interfaces have distinct non overlapping IP addresses? IP address and netmasks?
If they do, then I think that should cover "came in on interface A so reply goes out on interface A"
If things overlap it gets interesting. I don't think you would need specific rules to cover this (loadbalancers typically do this) but alot depends on specifics
 
I decided to keep digging by reading man pages just because I was curious, and discovered (much to my surprise) that FreeBSD has a SO_SETFIB socket option that can set a non-default FIB for any socket descriptor.

This means that a single process should be able to independently-use different default routes simultaneously, if:
1) Each FIB has been configured with the default route gateway for it's associated public IP
2) The code uses a separate socket descriptor bound to each public IP
3) The code calls setsockopt() after creating any sockets that need a non-default FIB; e.g.:
Code:
setsockopt(sock, SOL_SOCKET, SO_SETFIB, &fibnum, sizeof(fibnum))

This would only solve the OP's problem if willing to change server daemon code to add the appropriate setsocketopt() calls, but maybe others here will find it useful to know about SO_SETFIB.
 
you can use ipfw fwd to the respective gw on outgoing packets if their src matches the local addr you have from respective provider
that i used in the past and it worked
or probably ipfw setfib will work too
This is called policy routing.
 
ogogon : Considering all of the suggestions so far, I think the simplest-possible solution here is to use one ipfw fwd rule to force sending of any packets from your "secondary" IP back out your secondary interface.

I believe multiple FIBs would work, but it's more complex to setup vs adding one fwd rule.

Assuming your secondary public interface has IPv4 address "secIP", and the gateway associated with your secondary interface is "secGateway", then this ipfw rule should work:

Code:
ipfw add fwd secGateway all from secIP to not 10.0.0.0/8,172.16.0.0/12, 192.168.0.0/16

Details:
  • It's important to use secIP (vs trying to use the interface name) in the ipfw fwd rule.
  • If you already have other ipfw rules, additional ipfw rule changes may be required to "merge" your existing functionality, especially if you already have other ipfw rules that NAT-translate any traffic through your secondary interface. Let me know if you want assistance with that.
  • If your secondary interface IP is dynamic, then you'll need to update the ipfw fwd rule anytime secIP changes.
  • This solution should work fine for any TCP services listening on your secondary IP (even when they use a wildcard bind/listen).
  • In order to work properly, any UDP port(s) listening on your secondary interface will need to bind to the specific secondary-interface-IP instead of a wildcard-bind(), otherwise the kernel will select the primary-interface-IP as the source-address (the one implied by fib 0's default-route), because UDP is "connectionless", so the kernel simply doesn't/can't have the "context" to know that any given packet being sent is in-response to any earlier packet that was received on the secondary interface -- with UDP, only the application could know/track that.
  • FreeBSD system UDP services (e.g.: unbound and ntpd) appear to bind to each separate interface address, so that they can assure the proper source-address is put on any response packets.
  • The reason for the to not 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 in the fwd rule is to assure this policy-routing only applies only to clients out on the public Internet -- guessing you don't want replies for your internal/private hosts to be sent out to the public Internet.
  • When doing policy-based routing like this using ipfw fwd, it shouldn't be necessary to setup any routing in any non-zero FIBs.
Hope this sums it up clearly. Would be curious to hear what you actually did, and how well it worked.
 
Back
Top