Solved FIBs, Jails, and Routes: Not Working As Expected

[jasonvp]

Hey folks -

I have a server with a lagg interface of two 10GigE interfaces, and it's VLAN'd as well. On VLAN, 200, is my private, RFC1918 addressed internal VLAN. The second, VLAN 50, is the public side. What I'm attempting to do is put interface lagg0.50 into fib 1, and start public-facing jails in that fib and on that interface. The server itself doesn't have an IP on lagg0.50, it only does on lagg0.200 (the internal).

The thing I'm seeing is that host routes for each jail cross-bleed into the opposing fib. Or, more accurately, they get put into both fibs for some reason. For instance, my public DNS server is scarecrow. When I start his jail, his /32 gets added to fib 1, and it also gets added to fib 0. The reverse happens when I start my nas file serving jail. When he starts, his /32 gets put into both fibs: 0 and 1.

I can go through and manually remove all of the incorrect fib routes, but I have to do that every time a jail restarts. Which is tedious. Basically, what I want to prevent is a public jail directly accessing a private jail without routing through my upstream router first. With the host routes in each fib, that's exactly what happens: the public jail tries to reach directly to the private via the interface. And if I don't have pf running on the server, the jail will be able to do that.

Scarecrow's jail.conf entry (with the IP removed) is:

# scarecrow (external, authoritative DNS)
scarecrow {
        host.hostname = "scarecrow";
        exec.fib=1;
        ip4.addr += "lagg0.50|[REDACTED]/32";
}

The nas one is different of course:

# nas
nas {
        host.hostname = "nas";
        ip4.addr += "lagg0.200|192.168.10.2";
        mount = "/local/export /local/jails/nas/export nullfs rw 0 0";
        mount += "/local/timemachine /local/jails/nas/timemachine nullfs rw 0 0";
}

The appropriate bits in /etc/rc.conf

vlans_lagg0="50 200"
ifconfig_lagg0_50="up fib 1"

# lagg1 (private)
ifconfig_lagg0_200="inet 192.168.10.1/24"
defaultrouter="192.168.10.254"

And lagg0.50 is actually in fib 1 as expected:

#    ifconfig lagg0.50 | grep fib
    fib: 1

Is there a way to accomplish what I want? In other words, force the jails to follow their fibs' default route to get to the jails in the other fib, without manually deleting the /32s?

Thanks.

 

[Lamia]

I can go through and manually remove all of the incorrect fib routes, but I have to do that every time a jail restarts.

Use rc.local
Add/delete routes as desired into the default routing table via /etc/rc.local for the host machine. Using rc.local ensures that you don't have to manual do it at every reboot.

You may also need setup additional routes in your jail conf depending on what you are wanting to achieve. We needed to do so to get through to a jail from bhyve, and perhaps vice versa. But the host route is the most critical to change.

 

[Lamia]

Add to the fibs to use for each entry into rc.local.

Let's I forget, you may also need set PF rules for outgoing/incoming traffic to use specific fib(s).

 

[jasonvp]

Using rc.local is clumsy and sledge hammer like. I can do that, of course, but was hoping there was a simpler and easier way. And again, that doesn't help if I have to restart a jail for some reason. Every time I do that, I have to go and remove its route from the other fib.

 

[jasonvp]

I should have done a little more searching on this site. Looks like someone already ran into it as described here. I had that boot loader config set and thought it was supposed to do the right thing. Apparently I needed to set it with sysctl. Working now.

 

[Lamia]

I should have done a little more searching on this site. Looks like someone already ran into it as described here. I had that boot loader config set and thought it was supposed to do the right thing. Apparently I needed to set it with sysctl. Working now.

That's one of the requirements for using fibs/vlan. I was under the impression that you have done so long ago. It should be in sysctl.conf. What goes in loader.conf is how many fibs you want.

 

[jasonvp]

That's one of the requirements for using fibs/vlan. I was under the impression that you have done so long ago. It should be in sysctl.conf. What goes in loader.conf is how many fibs you want.

That's not true prior to 12.0. You could put that call in the loader.conf and it would do the right thing as expected. And that's what confused me; I thought it was doing the right thing, and it wasn't.