Thank you. I am much more interested in aspects that I have not found described anywhere.
For example. Let's assume that the connection was established via the ue0 interface, affiliated with the FIB2 routing table. How can I make sure that the response packets of this connection are also sent via the routing described in FIB2, and not FIB0? Is there such a possibility?
Once upon a time, when using the "FIB approach" to make a FreeBSD host "multi-homed", all one needed to do was:
- Set
net.fibs=N in /boot/loader.conf (where N is the number of desired routing tables; e.g.: 4).
- Add a
route add default 5.6.7.8 -fib 2 somewhere in the system startup (assuming that 5.6.7.8 is the gateway for your "secondary" Internet connection.
- Start a separate daemon for each service you want to offer to Internet clients -- one bound to your primary public IP, the other bound to your secondary public IP, and the daemon on your secondary IP started with with
setfib n; e.g.: setfib 2 /usr/local/etc/rc.d/yourdaemon.
One nuance here is that every process has a FIB# associated with it (can be viewed via
sysctl net.my_fibnum and set via
setfib n). And that FIB is inherited by any spawned-children.
But since Nov 8 2020 (src commit 2d39824195933c173bbfc9b31773070202d2e30e), the default for
sysctl net.add_addr_allfibs changed from 1 to 0, which means that non-zero FIBs no-longer automatically get
any routes for any of your local interfaces (other than lo0), so if you are using FIBs in the "modern world", you apparently need to either reset
sysctl net.add_addr_allfibs=1 before network startup (to revert to the "old" behavior), -OR- manually add interface routes for all of your relevant non-zero FIBs before you will be able to add a default route in one of your non-zero FIBs to your secondary gateway.
Here's an example how to add an interface route to a non-zero FIB:
route add -net 192.168.0.0/24 -iface vlan0 -fib 2
When
net.add_addr_allfibs=0, at least one local interface route must be configured in that FIB before a default route could be added to it.
If you're not certain your secondary FIB default-route is working properly, I recommend verifying it; e.g.:
setfib 2 traceroute 8.8.8.8
then see if the trace shows your packets going through your alternate gateway.
If you're running one or more services that were architected to do only one "wildcard" listen/bind and you expect only that one process to respond from all your public IPs, then the
setfib approach probably won't work cleanly, because the kernel (by default) will send
all outbound packets through the default gateway associated with that process's current FIB. Like SirDice said, there can be "only one" default route for any given process.
An implication if you don't configure this correctly would be that a packet with provider X's source-address might be sent out through provider Y (with your provider X source-address), but your provider Y won't get bidirectional packet-flow when provider X is down, probably circumvents the reason you are doing multi-homing. And some providers might blackhole packets being sent from their network if you put a source-address on an outbound packet that is
not within their network (i.e.: they might filter packets that seem "spoofed"). So configuring this correctly is required if you want it to work as-intended.
Bottom line: if you can run separate daemons (each bound to each of your public IPs), I think that's likely to work cleanly. But if you want only one daemon process to work with a wildcard-bind, then covacat's suggestion to use ipfw fwd rules probably is the best approach. FWIW, I have used ipfw fwd for this myself, it should work for TCP services, but if you have any UDP involved, then your daemon source code likely will need to be changed to open/bind one UDP socket per public IP, otherwise the daemon will not be able to select the desired/proper source address on outbound UDP packets.
To be clear, you probably wouldn't need to configure/use any FIBs if you use ipfw fwd, but like I said, that probably won't work cleanly if any UDP wildcard-binds are involved.
Let me know if I may be of any assistance with configuring ipfw for this..
