Routing issues on secondary FIB

T

therealbillhoff

I'm loosely trying to follow this guide on routing a jail through OpenVPN. Setting the jail portion aside for now as that's the easy part, I've got an issue with the routing for the secondary fib.

In that guide, a simple setfib 1 route add default 10.0.0.1 is used to set a default route for the secondary fib. It's likely 10.0.0.1 is an external router in that case. In my case, 10.0.0.1 is the machine where I'm trying to configure this. It is also the router for the entire network. If I add that route, I can't connect to outside sources as the routing table pretty much loops back on itself. For example:

Code: 
# setfib 1 route add default 10.0.0.1
add net default: gateway 10.0.0.1 fib 1

# setfib 1 ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.043 ms
^C
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.043/0.043/0.043/0.000 ms

# setfib 1 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
36 bytes from mimas.codexterous.hq (10.0.0.1): Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 24ce   0 0000  01  01 3f00 10.0.0.1  8.8.8.8
 
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

# setfib 1 traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  mimas (10.0.0.1)  0.043 ms  0.036 ms  0.024 ms
 2  mimas (10.0.0.1)  0.033 ms  0.030 ms  0.028 ms
 3  mimas (10.0.0.1)  0.027 ms  0.031 ms  0.032 ms
    ...snip...
64  mimas (10.0.0.1)  0.132 ms  0.138 ms  0.137 ms

The public interface on this machine is on another physical port. If I manually add the ISP provided gateway which is assigned via DHCP, the fib works - though that's not a long term solution as the gateway can change. I'm aware I may be able to set up some sort of hook to re-assign the gateway automatically when provided with a new one, but I'd rather do this the clean and correct way.

I guess what I'm looking for is how to route traffic on that fib over the other interface, rather than over an IP? Is that even a thing? I don't know enough about routing to really know where I'm meant to be going with this.

Here's an abbreviated version of the routing table on both fibs (some already established OpenVPN tunnels and things omitted):

Code: 
# netstat -r4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            lo0-100.NYCMNY-VFT UGS         em0
10.0.0.0/16        link#1             U           re0
10.0.0.0/8         172.17.10.25       UGS        tun2

# setfib 1 netstat -r4
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            this_server        UGS         re0
10.0.0.0/16        link#1             U           re0

I appreciate any help at all with this!
 
I think you'd better ask specific questions about that guide on that blog. I'm definitely not going to study all that ;)

Anyway, I can't help wonder why you'd even want to bother with a second routing table for all this? Seems like a lot of unnecessary overhead to me.

What is it you're trying to achieve here? Routing jail traffic over a specific interface, such as tun1? Because that should be doable even without any extra routing tables, depending on the way you set up your jail of course.
 
I'm not asking a specific question about that guide. At all. I am asking a specific question about routing on secondary FIBs. I offer a link to the guide as background informatino.

Ultimately I want to be able to route any given traffic of my choosing (based on a given subnet) over a VPN. The VPN changes the gateway of the entire machine so cannot run on the primary routing table.

But the question is not "should I use a secondary FIB for what I need to do", because that much is already known - I need a secondary routing table.

The question is "how can I get the routing to work on this secondary FIB".

Thank you for your time.
 
therealbillhoff said:
Ultimately I want to be able to route any given traffic of my choosing (based on a given subnet) over a VPN. The VPN changes the gateway of the entire machine so cannot run on the primary routing table.
Click to expand...
The usage of a different gateway doesn't automatically imply the need for 2 routing tables. And I can't help but wonder if this isn't also the cause of your problem.
 
ShelLuser said:
The usage of a different gateway doesn't automatically imply the need for 2 routing tables. And I can't help but wonder if this isn't also the cause of your problem.
Click to expand...

It actually does in this case. It's a bit more nuanced than that. OpenVPN entirely replaces the default gateway, causing the machine to not respond over the normal network. There isn't a way around that which keeps everything in place the way it needs to be. There is page after page of information about this all over the internet - it's quite a common thing. What I'm doing requires a secondary routing table. Additionally a second routing table is exactly the way that pfSense does exactly what I want to do.

But let's leave that completely aside, because it's not relevant at all. I am not interested in engaging in a discussion about why I should or shouldn't do what I know that I need to do. What is relevant is that I need the secondary network stack to be routable to the internet. That is the only thing in question here.

The thing holding that up, as demonstrated clearly in the original post, is that the machine I'm doing this with *is* the gateway. Various solutions for this simply use another gateway on the network to route the secondary network stack. This isn't possible in my case.

I'm just missing a little bit of routing magic (perhaps with pf, if not with route) to make this work. It really is extremely simple, and boils down to this very simple question: Routing the secondary network stack over 10.0.0.1 loops back and creates a routing loop.
 
You must log in or register to reply here.
Back
Top