fib/setfib question

Beeblebrox

Beeblebrox

Hello.

On my system I have Internal-Network, External-Network, lo0 and a cloned lo2 for jails. Traffic from lo0 and the Internal-Network for certain ports (like 80) will be diverted first to proxies running in jails and then to the outside (Ext-If). The other ports will forward requests to the gateway directly. It was suggested I use multiple routing tables for this instead of redirects in pf. I have read a good amount of documentation and get the concepts, but I have minor points to clear up.

  • The lo2 clone can use the 192.168.2.96/28 IP address group yet each jail is to have one of 192.168.2.(97-105)/32 address assignments. Do I set[ ]up one FIB for the lo2 address group (preferable but seems unlikely) or do I set one-FIB-per jail with
    Code: 
    jail_<name>_fib=n
    in jail.conf?
  • I assume I also need to assign one FIB to the Int-If NIC? If yes, how is it done persistently in /etc/rc.conf? I came across this code, but it does not seem very logical:
    Code: 
    setfib 1 route delete default
setfib 1 route add default 192.168.2.1 (Int-If's IP)
  • Same question as above, but for the jail. I would assume that
    Code: 
    jail_<name>_fib=n
    would take care of the whole thing.
  • What (if any) should be the defaultrouter= setting in <jail>/etc/rc.conf? Choices are: Nothing / The FIB address / The Ext-If address. It seems FIB address is the correct choice.

Thanks and regards.
 
@kpa: Thanks - that would make sense of course.

That takes care of Q4. For Q2, I came across information which suggests creating a custom rc script. The example was, set
Code: 
vlan200_enable=YES
and place in /usr/local/etc/rc.d/vlan200:

Code: 
#!/bin/sh

# PROVIDE: sumtel-watchdog
# REQUIRE: DAEMON
# KEYWORD: shutdown
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf to
# enable vlan200:
# vlan200_enable (bool): Set to NO by default.  Set it to YES to  enable vlan200.

. /etc/rc.subr
name="vlan200"
rcvar="vlan200_enable"
pidfile="/var/run/${name}.pid"

start_cmd=${name}_start
stop_cmd=${name}_stop

vlan200_start() {
  /usr/sbin/setfib 2 /sbin/dhclient vlan200
}

vlan200_stop() {
  /bin/pgrep -f "dhclient: vlan200" | /usr/bin/xargs kill
}

load_rc_config $name
: ${vlan200_enable="NO"}
run_rc_command "$1"

This is for DHCP assigned address, but illustrates the concept of what to do nicely I think. There's also this thread, but the concept does not seem implemented (cannot find documentation).

Lastly, I remembered a Q5 which I had forgotten to ask: how does one set rules for routing tables in pf? From what I have read, pf can be given rules based on each fib, but the examples I found were a bit ugly. Does pf treat each fib like a device (like Ext_If for example)?
 
Last edited by a moderator:
Update: I decided to test the "one-FIB-per-jail" approach. The correct syntax for that in /etc/jail.conf is
Code: 
exec.fib = n;
The jail starts normally and I now have two fibs. The table for fib1 shows the gateway for each network class by device (re0, re1), but there is no "default gateway", meaning how will traffic from jails know to exit from ExtIf to the Internet?

My second observation is that fib1 has routing information on the two other jails (.99 and .100) even though no fib has been set up for them. I suppose that's due to all jails residing on cloned device lo2? Perhaps I don't have to create one-FIB-per-jail after all?

Code: 
 # setfib 1 netstat -rn
Routing tables (fib: 1)
Internet:
Destination        Gateway            Flags    Netif Expire
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.2.0/26     link#2             U         re1
192.168.2.97       link#4             UH        lo2
192.168.2.99       link#4             UH        lo2
192.168.2.100      link#4             UH        lo2

#  netstat -rn
Routing tables
Internet:
Destination        Gateway            Flags    Netif Expire
default            192.168.1.1        UGS       re0
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.1.10       link#1             UHS       lo0
192.168.2.0/26     link#2             U         re1
192.168.2.1        link#2             UHS       lo0
192.168.2.97       link#4             UH        lo2
192.168.2.99       link#4             UH        lo2
192.168.2.100      link#4             UH        lo2
 
There's a sysctl to prevent automatic adding of all the directly connected interfaces: echo 'net.add_addr_allfibs=0' >> /boot/loader.conf. You'll need to specify your routes in your rc.conf manually. Something like this can give you an idea.

Code: 
ifconfig_em0_alias0="inet 192.168.102.11/32 fib 2"
static_routes="fibnetwork fibdefault"
route_fibnetwork="-net 192.168.102.0/24 -interface em0 -fib 2"
route_fibdefault="default 192.168.102.1 -fib 2"

Oh, and I hope you're not burning time compiling a custom kernel just for FIBs, it can be set at boot time on GENERIC: echo 'net.fibs=4' >> /boot/loader.conf.
 
Hi @junovich, thanks for the answer.

There's a sysctl to prevent automatically added all the directly connected interfaces: echo 'net.add_addr_allfibs=0' >> /boot/loader.conf. You'll need to specify your routes in your rc.conf manually...
Oh, and I hope your not burning time compiling a custom kernel just for FIBs, it can be set at boot time on GENERIC: echo 'net.fibs=4' >> /boot/loader.conf.
Click to expand...
Yep, I already had that part solved.

Looking through your /etc/rc.conf suggestion, I did as below, but I probably did something wrong:
Code: 
network_interfaces="re0 re1 lo0"
cloned_interfaces="lo2"
ifconfig_lo0="inet 127.0.0.1"
ifconfig_re0="inet 192.168.1.10/24 fib 0"
ifconfig_re1="inet 192.168.2.1/26 fib 1"
ifconfig_lo2="inet 192.168.2.97/28 fib 2"
static_routes="internal jail default"
route_jail="-net 192.168.2.96/28 -interface lo2 -fib 2"
route_internal="-net 192.168.2.0/26 -interface re1 -fib 1"
route_default="default 192.168.1.1 -fib 0"
#defaultrouter="192.168.1.1"
Code: 
 # netstat -rn
Destination        Gateway            Flags    Netif Expire
default            192.168.1.1        UGS       re0
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.1.10       link#1             UHS       lo0
192.168.2.0/26     link#2             U         re1
192.168.2.1        link#2             UHS       lo0
192.168.2.97       link#4             UH        lo2

# setfib 0 netstat -rn
Destination        Gateway            Flags    Netif Expire
default            192.168.1.1        UGS       re0
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.1.10       link#1             UHS       lo0
192.168.2.0/26     link#2             U         re1
192.168.2.1        link#2             UHS       lo0
192.168.2.97       link#4             UH        lo2

# setfib 1 netstat -rn
Destination        Gateway            Flags    Netif Expire
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.2.0/26     link#2             U         re1
192.168.2.97       link#4             UH        lo2

# setfib 2 netstat -rn
Destination        Gateway            Flags    Netif Expire
127.0.0.1          link#3             UH        lo0
192.168.1.0/24     link#1             U         re0
192.168.2.0/26     link#2             U         re1
192.168.2.96/28    lo2                US        lo2
192.168.2.97       link#4             UH        lo2

This setup also does not solve the next problem which will come up: When I start jail #1 with IP 192.168.2.97/32, the IP assigned to lo2 will get destroyed as will the /28 network stack assigned to it, thus leaving fib2 incapacitated.
 
Last edited by a moderator:
The silence is deafening! OK, let me ask a part of the question in a different manner: When using fib with jails, does each jail need its own fib, i.e. jail-1 IP=192.168.2.97/32 and it must have its own fib since address range is 1? Or,can I get away with using only ONE fib for several jails if I am able to assign an IP with /28 address range to the fib, then assign that address as the gateway to all the jails?
 
The way I use it, I have one fib for multiple LAN services and one fib for DMZ services. Both have multiple jails on each with multiple aliases tied to each physical NIC and a /24 subnet on each side. I haven't tried it with a cloned loopback. When I have more than just my phone in front of me I can probably look at this a bit closer.
 
How are you starting your jails? Are you using the form lo2|192.168.2.97/28 or just 192.168.2.97/28 in your jail.conf?
 
The way I use it, I have one fib for multiple LAN services and one fib for DMZ services ... I haven't tried it with a cloned loopback.
Click to expand...
That's what I was thinking as well, when I assigned re0, re1 and lo2 each their own fib. Cloned loopback should make no difference IMHO.

How are you starting your jails?
Click to expand...
I'm using /32 for the jail IP's, while using /28 for fib2. As stated in post #6, I only assign the address range to fib2 but no IP:
Code: 
jail1 {
interface = lo2;
ip4.addr = 192.168.2.97/32;
exec.fib = 2;     }

jail2 {
interface = lo2;
ip4.addr = 192.168.2.98/32;
exec.fib = 2;     }

Are you using the form lo2|192.168.2.97/28 or just 192.168.2.97/28 in your jail.conf?
Click to expand...
That, in essence is my question: /28 fib but /32 jails on that fib interface, AND how do you assign the gateway address in that scenario? My guess is that gateway with 192.168.2.97/28 cannot co-exist with a jail on 192.168.2.97/32 and the gateway will have to be assigned a separate address on the /28 subnet. Unless of course 192.168.2.96/28 can be assigned as gateway - which would solve the problem.
 
I suspect that since you have assigned 192.168.2.97 on the host with a /28, that attempting to assign the jail as a /32 is causing it to blow away that interface. Have you tried setting your jails to .98 and .99 with a /32 subnet? That should assign them as proper alias address. Also, you cannot use .96 or .111 as they are reserved for the network and broadcast within that subnet.

To expand on what I do, see below. I tell my jails how to reach their local subnet and give them a default router of an external device. As far as my jails are concerned, they do not route through the FreeBSD box and for a LAN jail to talk to a DMZ jail, it ends up sending packets out one NIC to a router only to have those packets come back down another wire on a different VLAN to the other NIC. Without the FIB configuration, one jail would talk to the other via the loopback interface and those packets would never leave the FreeBSD machine.

Code: 
static_routes="dmzfibnetwork dmzfibdefault lanfibnetwork lanfibdefault"
route_dmzfibnetwork="-net 192.168.102.0/24 -interface em0 -fib 2"
route_dmzfibdefault="default 192.168.102.1 -fib 2"
route_lanfibnetwork="-net 10.100.102.0/24 -interface em1 -fib 1"
route_lanfibdefault="default 10.100.102.1 -fib 1"
 
Thank you for your help. I found a way to solve the problem without using fib, and using redirect in pf.conf instead. I hope to post the full solution in a later thread.

BTW; you were apparently right about your "cloned interfaces" comment. from the man page for ifconfig:
The FIB is not inherited, e.g., vlans or other sub-interfaces will use the default FIB (0) irrespective of the parent interface's FIB.
Click to expand...
 
I'm glad you got it worked out. I found that some combinations of reply-to/route-to statements caused kernel panics and found the fib solution to work well for what I was looking for instead. I would be interested in seeing the pf.conf solution that you worked out.
 
You must log in or register to reply here.
Back
Top