Solved Two network interfaces and default routes

K

karihre

Hello,

I've fairly recently stepped into the FreeBSD world after being a long time Linux user. I'm running into a problem that I've managed to solve in my Linux servers but I can't seem to figure it out on the FreeBSD machines, running FreeBSD 10.1.

So the servers all have two interfaces, one has ip 10.4.32.12 on the network 10.4.32.0/25 and the other has ip 192.168.16.12 on the network 192.168.16.0/20. On each network there are routers (10.4.32.1 and 192.168.16.1) that connect to other networks, currently the default route is 192.168.16.1.

The problem arises when I connect to the 10.4.32.12 address from the outside (another 10. network), I successfully reach the server, but it's replies go out the default route to 192.168.16.1 and never reach me, I can see the incoming packets on the 192.168.16.1 router.

So how do I configure FreeBSD to send traffic from ip address 10.4.32.12 through the default gw 10.4.32.1 ? On my Linux boxes I would set up 2 routing tables, one for each default gateway, then I can say:
ip rule add from 10.4.32.12 table alternate_routing_table

I've seen quite a few threads on these forums, but for some reason I can't seem to get the suggestions to work for me. I recompiled the kernel to support multiple routing tables (options ROUTETABLES=16). Relevant configurations are as follows:
Code: 
root@storage:~ # cat /boot/loader.conf
ipfw_load="YES"
net.fibs=16
root@storage:~ # sysctl net.fibs
net.fibs: 16
Code: 
root@storage:~ # cat /etc/rc.conf # cut out irrelevant things
ifconfig_vmx0="inet 192.168.16.12 netmask 255.255.240.0"
ifconfig_vmx1="inet 10.4.32.12 netmask 255.255.255.128"
defaultrouter="192.168.16.1"
firewall_enable="YES"
firewall_script="/etc/rc.firewall"
firewall_type="open"
Code: 
root@storage:~ # cat /etc/rc.local
#!/bin/sh

setfib 0 route delete default
setfib 0 route add default 192.168.16.1
setfib 1 route delete default
setfib 1 route add default 10.4.32.1

ipfw -f flush
ipfw add allow   all from any to any via lo0
ipfw add setfib 1 ip from any to any via vmx1
ipfw add setfib 0 ip from any to any via vmx0
ipfw add allow    ip from any to any
Code: 
root@storage:~ # setfib 0 netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.16.1       UGS        vmx0
10.4.32.0/25       link#2             U          vmx1
10.4.32.12         link#2             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.16.0/20    link#1             U          vmx0
192.168.16.12      link#1             UHS         lo0

root@storage:~ # setfib 1 netstat -rn4
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags      Netif Expire
default            10.4.32.1          UGS        vmx1
10.4.32.0/25       link#2             U          vmx1
127.0.0.1          link#3             UH          lo0
192.168.16.0/20    link#1             U          vmx0
Related threads:
- http://daemonforums.org/showthread.php?t=4610
- https://forums.freebsd.org/threads/...d-routing-ipfw-configuration-for-jails.42112/
- https://forums.freebsd.org/threads/two-default-routes-and-vlans.37900/

Am I missing something obvious? Any hints are appreciated! Thank you.
 
Unfortunately I cannot help much with the ipfw(8) portion as I mainly use pf(4).

karihre said:
So how do I configure FreeBSD to send traffic from ip address 10.4.32.12 through the default gw 10.4.32.1 ? On my Linux boxes I would set up 2 routing tables, one for each default gateway, then I can say:
ip rule add from 10.4.32.12 table alternate_routing_table
Click to expand...

The biggest difference here is that FreeBSD doesn't have the same underlying concept of sending from address x.x.x.x to destination y.y.y.y. Instead it's processes that must be explicitly bound to the FIB that you want them to run on. If you want the same service on multiple FIBs then multiple instances of the same service must be run. For example, if you want SSHD to listen on a management network on FIB 1, then you would use an /etc/rc.conf entry like this:
Code: 
sshd_fib="1"

karihre said:
I've seen quite a few threads on these forums, but for some reason I can't seem to get the suggestions to work for me. I recompiled the kernel to support multiple routing tables (options ROUTETABLES=16). Relevant configurations are as follows:
Code: 
root@storage:~ # cat /boot/loader.conf
ipfw_load="YES"
net.fibs=16
root@storage:~ # sysctl net.fibs
net.fibs: 16
Click to expand...
The custom kernel used to be the case years ago. I believe GENERIC now in all supported releases can be set at boot time via the /boot/loader.conf entry. The entry you already have will do that without the custom kernel. EDIT: I checked and the boot time ability to set FIB count was in the tree in early 2012. See https://www.freebsd.org/news/status...ulti-FIB:-IPv6-Support-and-Other-Enhancements
 
karihre said:
The problem arises when I connect to the 10.4.32.12 address from the outside (another 10. network), I successfully reach the server, but it's replies go out the default route to 192.168.16.1 and never reach me, I can see the incoming packets on the 192.168.16.1 router.
Click to expand...
Can you be a bit more specific as to what you're sending? Are we talking ICMP or daemon traffic?

If it is traffic towards a daemon Junovitch is correct in that the daemon can only live in one fib. SSHD in fib 0 might explain what you're seeing.
 
Hi and thanks for the answers.

I'm talking about daemon traffic, so from an outside network, if I ssh to 10.4.32.12 (one of the addresses of the server), this gets routed through the 10.4.32.1 router, but the reply packets (tcp ack i guess) from sshd are sent towards the 192.168.16.1 router since the source address of my initial connection is not directly on the "adjacent" 10.4.32.0/25 network. Seeing as the 192.168.16.1 router doesn't know the way back to me the packets are effectively dropped.

So from reading your answers I did the following. I modified /etc/ssh/sshd_config to listen only on the 192.168.16.12 ip address, I then copied it to /etc/sshd/sshd_config.10 and changed that file to set the listen address to 10.4.32.12, I then edited /etc/rc.local to contain the following:
Code: 
#!/bin/sh

setfib 1 route delete default
setfib 1 route add default 10.4.32.1

setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.10 # listens on 10.4.32.12
My thinking is then that the normal boot process will start the primary sshd that defaults to fib 0 and uses the main sshd config file. Then in /etc/rc.local I start the alternate sshd process that listens on 10.4.32.12. This seems to be working, so thanks are in order! Is there a better (more FreeBSD) way to configure this or this is good enough?

Thanks for the replies!
 
karihre said:
Hi and thanks for the answers.

I'm talking about daemon traffic, so from an outside network, if I ssh to 10.4.32.12 (one of the addresses of the server), this gets routed through the 10.4.32.1 router, but the reply packets (tcp ack i guess) from sshd are sent towards the 192.168.16.1 router since the source address of my initial connection is not directly on the "adjacent" 10.4.32.0/25 network. Seeing as the 192.168.16.1 router doesn't know the way back to me the packets are effectively dropped.

So from reading your answers I did the following. I modified /etc/ssh/sshd_config to listen only on the 192.168.16.12 ip address, I then copied it to /etc/sshd/sshd_config.10 and changed that file to set the listen address to 10.4.32.12, I then edited /etc/rc.local to contain the following:
Code: 
#!/bin/sh

setfib 1 route delete default
setfib 1 route add default 10.4.32.1

setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.10 # listens on 10.4.32.12
My thinking is then that the normal boot process will start the primary sshd that defaults to fib 0 and uses the main sshd config file. Then in /etc/rc.local I start the alternate sshd process that listens on 10.4.32.12. This seems to be working, so thanks are in order! Is there a better (more FreeBSD) way to configure this or this is good enough?

Thanks for the replies!
Click to expand...

Is this the correct way to set your default routes on your fibs? I'm in the same situation so I'll most likely use your code example but I thought rc.local was deprecated for rc.conf or rc.d?
 
junovitch@ said:
There are examples of how to do default routes with FIBS through /etc/rc.conf in different threads across the forum. For example:
https://forums.FreeBSD.org/threads/multiple-ips-in-different-subnets-default-routing.50699/
Click to expand...

I actually figured it out and even created a howto for my particular setup:
https://forums.freebsd.org/threads/...-route-tables-for-each-jail-setfib-fib.53944/

rc.local is NOT needed - everything OP did in rc.local can be done cleaner in rc.conf
 
You must log in or register to reply here.
Back
Top