Do I need to recompile kernel to use FIBs...?

[Deleted member 79566]

I'm getting mixed results online. Some sites say I do and some don't mention it. Problem is I didn't install the source... nor a compiler... heck I don't even have make installed. So I'm trying to avoid figuring all that out if I can. I just need to use FIBs for isolating network routes. I'm at a point where I can see traffic on both interfaces in the FIB via Wireshark but traffic isn't going anywhere outside the system... I'm at a loss trying to figure this out. $5 says it's some dumb crap like forgetting to set net.inet.ip.forwarding. Actually it feels a lot like forgetting that... but it's not that

 

[covacat]

fibs work with generic kernel you just have to set the max number in sysctl.conf

 

[sko]

IIRC the "ROUTETABLES" kernel option has been deprecated sometime along (or with the release of?) the 12 branch and multiple FIBs enabled by default. Their number can be set via the net.fibs sysctl.
A lot has changed in terms of routing since then. E.g. the routing stack received a major rewrite for the 13.0-RELEASE. So I'd be *very* careful following any source that still mentions recompiling the kernel - it might be heavily outdated in more than one aspect!

TBH the handbook is really lacking some parts in regards of routing, the setfib(1) manpage doesn't go into much detail and ifconfig(8) even still has outdated information about the kernel option...
I only got my head around FIBs enough to confidently use them in production, when I started exploring and using routing domains on OpenBSD - they basically had the same idea but followed it completely through. FIBs on FreeBSD in comparison feel a bit like they only went "half the way", but they also get the job done for most scenarios.

But regarding your problem:
- I assume you specified a common fib for the interfaces you want to segragate in the same group? (show output of ifconfig)
- you added routes to the FIB so that network view actually knows where to send packets to? ( netstat -nrF<FIB>)
- there is currently no firewall running or you have allowed traffic in that view/FIB to flow freely through the system?

 

[Deleted member 79566]

Yeah I had a feeling I was following tutorials from mismatching versions. I left the kernel compiling overnight but I guess I don't need to install it now.

I have the two interfaces in the same FIB, yes. There's a second FIB with a third interface (which is the system default route), but that one gets its configuration from DHCP on the other end, and should start working on its own as soon as the other two are fixed.

side note: I just spent an unspeakable amount of time trying to figure out how to mount a simple FAT32 usb stick just for getting these outputs out of the machine. The mount command here is hardcore.

Here's the outputs of the commands you mentioned. I took out the MAC addresses and edited my public IP to 64.64.64.64 and my ISP's gateway to 64.64.64.1. Also, the two interfaces that have both an `ether` and a `hwaddr` is because I changed the MACs in rc.conf. As for the firewall, I have pf configured but I have the service disabled at the moment.

ifconfig:

nfe0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=82008<VLAN_MTU,WOL_MAGIC,LINKSTATE>
    ether [[redacted]]
    hwaddr [[redacted]]
    inet 192.168.1.153 netmask 0xffffff00 broadcast 192.168.1.255
    inet6 fe80::4441:47ff:fe47:3054%nfe0 prefixlen 64 scopeid 0x1
    fib: 1
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether [[redacted]]
    inet 4.2.2.1 netmask 0xffffff00 broadcast 4.2.2.255
    inet6 fe80::527c:6fff:fe52:ba28%igb0 prefixlen 64 scopeid 0x2
    fib: 2
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether [[redacted]]
    hwaddr [[redacted]]
    inet 64.64.64.64 netmask 0xffffff00 broadcast 255.255.255.255
    inet6 [[redacted]]%igb1 prefixlen 64 scopeid 0x3
    fib: 2
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
    options=0
    groups: pflog

netstat -rnF2 # (the FIB with two interfaces)

Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            64.64.64.1        UGS        igb1
4.2.2.0/24         link#2             U          igb0
4.2.2.1            link#4             UHS         lo0
64.64.64.0/24      link#3             U          igb1
64.64.64.64        link#4             UHS         lo0
127.0.0.1          link#4             UHS         lo0
192.168.1.0/24     link#2             US         igb0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#4                        URS         lo0
::1                               link#4                        UHS         lo0
::ffff:0.0.0.0/96                 link#4                        URS         lo0
fe80::%lo0/10                     link#4                        URS         lo0
ff02::/16                         link#4                        URS         lo0

netstat -rnF1 # (the FIB with one interface)

Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS        nfe0
127.0.0.1          link#4             UHS         lo0
192.168.1.0/24     link#1             U          nfe0
192.168.1.153      link#4             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#4                        URS         lo0
::1                               link#4                        UHS         lo0
::ffff:0.0.0.0/96                 link#4                        URS         lo0
fe80::%lo0/10                     link#4                        URS         lo0
ff02::/16                         link#4                        URS         lo0

I also have the following in /etc/rc.conf:

gateway_enable=yes
ipv6_gateway_enable=yes

And the following in /boot/loader.conf:

net.fibs=4
net.add_addr_allfibs=0

Can you tell where I screwed up?

 

[sko]

There's a second FIB with a third interface (which is the system default route)

I usually leave the mgmt interface (that also holds the default route for the host) in the default FIB (i.e. no special FIB configuration) and only put routing interfaces in a distinct FIB.
Multiple FIBs are usually only needed if you want/have to do some special routing with different default routes - e.g. if you have to route some traffic via a VPN - or if you want to isolate the mgmt interface of the host from the interfaces that take part in the actual routing.

I assume your local net is the '4.2.2.0/24' subnet?
Are you able to ping your default gateway (and some host on the internet) as well as some host in that private subnet?
The 192.168.1.0/24 route via igb0 in FIB2 will never work - that subnet is not on igb0. To route between FIBs you need firewall rules in place (is there any firewall, e.g. PF running? if yes show the config file for that) - again: this is only necessary if you have to route via different default routes. If 192.168.1.0/24 is just another local network that should be able to reach the internet (via igb1) just put it in the same FIB as the other two interfaces - and then there's no need to use FIBs as you don't need distinct routing tables or segragated network views.

Also, igb0 only connects at 100Mbit - is that intentional? Otherwise this might indicate a cabling problem...

side note: I just spent an unspeakable amount of time trying to figure out how to mount a simple FAT32 usb stick just for getting these outputs out of the machine.

Just ssh into the host and copy&paste from your terminal window. Or you can use termbin.com: ifconfig | nc termbin.com 9999; then post the shortlink for long output or copy&paste whats there.

 

[Deleted member 79566]

I assume your local net is the '4.2.2.0/24' subnet?

4.2.2.1 and 4.2.2.2 are made up addresses, since neither end of that connection has DHCP. I assume that's what one does when DHCP is not available? Or was I supposed to do something else? That subnet only has the two machines in question. My local net is on 192.168.1.0/24.

Are you able to ping your default gateway (and some host on the internet) as well as some host in that private subnet?

Assuming I prefix network commands with `setfib 2`, I can reach the internet no problem, but not the private subnet. It doesn't respond to ping nor telnet on ports that are supposed to be open. I've been trying to debug that end as well. I suspect I indeed was "supposed to do something else" about the lack of DHCP. That machine works just fine when plugged up to DHCP, but apparently not when set with a static IP. In fact if you don't see anything wrong with my config, it's probably a problem with the 4.2.2.2 machine... I may just resort to setting up a DHCP server on the FreeBSD side if nothing else works.

What's baffling though, is when I ping an external IP from a local machine (coming from igb0 aka 4.2.2.2) the ping packet shows up in Wireshark on both igb0 and igb1 even though it doesn't go anywhere after that. So there's at least somewhat of a connection established.

The 192.168.1.0/24 route via igb0 in FIB2 will never work - that subnet is not on igb0.

That's a mistake actually. I don't know how that ended up there. Indeed 192.168.1.0/24 belongs in FIB 1. I don't need the two FIBs to be crossing.

Also, igb0 only connects at 100Mbit - is that intentional? Otherwise this might indicate a cabling problem...

Ah, that must've been the crossover cable I was testing out. I've since switched back to a normal cable that reports 1000mbit. Sadly this was the only noticeable difference between the two cables. (The only legit 100mbit interface here is nfe0.)

Just ssh into the host and copy&paste from your terminal window. Or you can use termbin.com: ifconfig | nc termbin.com 9999; then post the shortlink for long output or copy&paste whats there.

Is that a pastebin for terminals...? No way. That's amazing. I'll use that next time.