Solved FreeBSD 14 and route in non-zero FIB

alexey

alexey

Hello,

I have tried to use the FreeBSD questions mailing list and did not receive any answers, so I am trying my luck here. Sorry for spamming.

After upgrading to FreeBSD 14 I have the following problem (it worked in 13.2). I wonder what am I doing wrong.

Code: 
# route add -fib 0 default 192.168.0.1
add net default: gateway 192.168.0.1 fib 0
# route add -fib 1 default 192.168.0.1
add net default: gateway 192.168.0.1 fib 1: Invalid argument

So route cannot be added with FIB 1. The FIB 1 does exist ( net.fibs=2
in /boot/loader.conf) or the error message would be different:

Code: 
# route add -fib 2 default 192.168.0.1
route: invalid fib number: 2
 
maybe it helps that you can now set default routes for FIBs directly via rc.conf:
It is now possible to add default routes for FIBs other than the primary by using the defaultrouter_fibN and ipv6_defaultrouter_fibN rc.conf(5) variables. 30659d1dcbcc (Sponsored by ScaleEngine Inc.)
Click to expand...


Regarding your specific error: do you actually have an active interface in FIB 1 with an address in the same subnet as that router? Otherwise setting a route will fail, regardless of the selected FIB:
Code: 
# netstat -nrF1
Routing tables (fib: 1)
# route add -fib 1 default 10.50.66.1
add net default: gateway 10.50.66.1 fib 1: Invalid argument

# ifconfig em0.666 create 10.50.66.99/24 fib 1
# netstat -nrF1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
10.50.66.0/24      link#5             U       em0.666
10.50.66.99        link#2             UHS         lo0
# route add -fib 1 default 10.50.66.1
add net default: gateway 10.50.66.1 fib 1
# netstat -nrF1
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.50.66.1         UGS     em0.666
10.50.66.0/24      link#5             U       em0.666
10.50.66.99        link#2             UHS         lo0
 
What I posted is a minimal example. Actually, I have a script that runs when OpenVPN goes up, and the router address is dynamic. It started to fail after the upgrade.

"do you actually have an active interface in FIB 1" -- that could be a culprit, thanks! I will check later. I see that you have fib setting in ifconfig -- is it something new?
 
On 13.2 I did not need fib in ifconfig:

Code: 
# netstat -rnF 1 | grep -q tun0 && echo "There is a route."
There is a route.
# ifconfig tun0 | grep -q fib || echo "No fib."
No fib.

If I create the interface as you wrote, there is fib:

Code: 
# ifconfig em0.666
em0.666: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4000403<RXCSUM,TXCSUM,LRO,NOMAP>
        ether 68:05:ca:c6:35:5a
        inet 10.50.66.99 netmask 0xffffff00 broadcast 10.50.66.255
        groups: vlan
        vlan: 666 vlanproto: 802.1q vlanpcp: 0 parent interface: em0
        fib: 1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=69<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL,NO_RADR>
# ifconfig em0.666 | grep -q fib || echo "No fib."
<nothing>
 
alexey said:
I have a script that runs when OpenVPN goes up, and the router address is dynamic.
Click to expand...
I'd check if that interface is actually up *and* has its address set before that route add command is triggered. Maybe you need to add a check for that first.
TBH I've fiddled way too much with scripts for bringing up more-or-less complex network configurations. Lesson learned: use the "default" mechanisms if possible - i.e. let OpenVPN handle adding the route (if possible with FIBs, haven't used OpenVPN for years). If the VPN opens a layer 2 tunnel, just let DHCP handle pushing the routes (maybe with overrides/additions via dhclient.conf(5) if needed)


alexey said:
I see that you have fib setting in ifconfig -- is it something new?
Click to expand...
Not sure how "new" this is, I've been using that for quite a while (at least since 12.X IIRC). It does almost the same as 'rdomain' in OpenBSD and puts all traffic from that interface into the given FIB (or routing domain on OpenBSD).
I'm using this mainly for routers, where I want the interfaces and FIB for routing completely separated from the host's management interface and default routing table. OpenBGPd then also runs in the context of the FIB/rdomain of the routing interfaces.
This also heavily simplifies the pf.conf as there is no need to specify the FIBs, except if you *really* want to route packets between different fibs/rdomains (headache alert!)
 
rdomain reference was really useful. Of course, I do check that the interface is up and the address is set, but maybe the time to rethink OpenVPN-FIB usage has come.
 
I think I had a related problem.

Code: 
exec.prestart+="ifconfig tun create";
exec.prestart+="/sbin/ifconfig tun0 inet 10.2.1.2 10.2.1.2 netmask 255.255.255.0 broadcast 10.2.1.255";
exec.prestart+="/sbin/route add -net 10.2.1.0 10.2.1.2 255.255.255.0 -ifp tun0";

In my openvpn jail I was unable to add a route after 14-RELEASE update.

First, the invalid argument was solved by going from the form
Code: 
ifconfig tun0 inet 10.2.1.2 netmask 255.255.255.0 broadcast 10.2.1.255
to
Code: 
ifconfig tun0 inet 10.2.1.2 10.2.1.2 netmask 255.255.255.0 broadcast 10.2.1.255

Then I had to switch everything from using 10.2.1.1 to 10.2.1.2.
Routes don't add to tun interfaces on .1 anymore for some reason. They go to loopback instead.

On openvpn I had to switch from the server 10.2.1.0 255.255.255.0 shortcut to doing the full set of commands.
Code: 
topology subnet
###################################################
# the next line is a shortcut for all the settings
#   to the end of this block.
#   server 10.2.1.0 255.255.255.0
#
# tun lost the ability to come up on .1 when
#   we went to 14-RELEASE. if you use 'server'
#   everything is set to use .1, so we had to
#   list the settings out explicitly instead.
#   we went to .2 and everything worked.

mode server
tls-server
push "topology subnet"
ifconfig 10.2.1.2 255.255.255.0     # set server ip
ifconfig-pool 10.2.1.4 10.2.1.253   # set client pool range
push "route-gateway 10.2.1.2"       # push server ip to clients
route-gateway 10.2.1.2              # set route gateway
###################################################
 
Hello,

I've managed to quickly solve the route problem with ifconfig <interface> fib 1 in my script, thanks! Now I have another problem: reproducible kernel panic with setfib 1 w3m -6 .... I guess I'll stay on 13.2 for the time being, and fill the bug if I find time for investigations. I guess this case is closed, thanks again!
 
Hi, I have similar issue, openvpn on FreeBSD 14.0. Works good on 13.2. Crash on initiate VPN from any clients. I don't add route. I received only message like:
Connection Attempt Failed to poll for packets: Operation not permitted (errno=1)
 
ve2tmq said:
Hi, I have similar issue, openvpn on FreeBSD 14.0. Works good on 13.2. Crash on initiate VPN from any clients. I don't add route. I received only message like:
Connection Attempt Failed to poll for packets: Operation not permitted (errno=1)
Click to expand...

Just upgraded to freebsd14.1 - experiencing same issue, openvpn crashing on connection initialize ...
 
You must log in or register to reply here.
Back
Top