Missing default gateway in multi-fib DHCP env

Lucian

Lucian

tl;dr - Second NIC doesn't get a def router via DHCP breaking multiple routing table setup

Hello,

I need to setup a VM with 2 NICs on DHCP, we need a separate routing table (fib) per NIC, but I'm not getting the default router for the second one.

What I've done so far:

In /boot/loader.conf:
Code: 
net.fibs=8

In /etc/sysctl.conf:
Code: 
net.add_addr_allfibs=0

And in /etc/rc.conf the following:
Code: 
ifconfig_em0="SYNCDHCP fib 0"
ifconfig_em1="SYNCDHCP fib 1"

However, when I check the routing tables, fib 1 is missing the default gateway.

Code: 
[root@freebsd-13_1 ~]# setfib 0 netstat -rn
Routing tables
Internet:
Destination        Gateway            Flags     Netif Expire
default            10.2.1.1           UGS         em0
10.2.1.0/24        link#1             U           em0
10.2.1.70          link#1             UHS         lo0

[root@freebsd-13_1 ~]# setfib 1 netstat -rn
Routing tables (fib: 1)
Internet:
Destination        Gateway            Flags     Netif Expire
10.13.13.0/24      link#2             U           em1
10.13.13.88        link#2             UHS         lo0
I thought it odd, definitely not what I needed or expected.. Initially I thought it was a fib bug, but then thought to check dhclient, so I added an exit hook looking like this:
Code: 
printenv >> /tmp/dhcplog

Output here https://pastebin.com/raw/uScDH15u

..And sure enough, looking into /tmp/dhcplog reveals em1 is not sent a "new_routers" option at all, only em0 gets it. I can confirm the network on em1 does work correctly and when reversing the NICs it does supply a valid router.

I even tried to "force" dhclient to request a router for em1 by configuring the interfaces expressly and _required_ the "routers" option as per this example ... without success.

Any more suggestions?
 
What if you change order of receiving IP and first will be em1? Which interface will not receive default gateway? Also, you should log this with hook "printenv" to see, what happens
 
skeletor changed rc.conf to this, it made no difference, as expected.
Code: 
ifconfig_em1="SYNCDHCP fib 1"
ifconfig_em0="SYNCDHCP fib 0"

Then changed it to this, and none of the fibs got a default route:
Code: 
ifconfig_em0="SYNCDHCP fib 1"
ifconfig_em1="SYNCDHCP fib 0"

The hook log is here https://pastebin.com/raw/uScDH15u
 
Code: 
# cat /etc/rc.conf.d/dhclient
unset dhclient_fib
[ "$ifn" = "wlan0" ] && dhclient_fib=1
ugly hack but... change wlan0 to whatever it suits
 
covacat said:
Code: 
# cat /etc/rc.conf.d/dhclient
unset dhclient_fib
[ "$ifn" = "wlan0" ] && dhclient_fib=1
ugly hack but... change wlan0 to whatever it suits
Click to expand...
Thanks, but no dice.
The problem is seemingly not with the setting of the fib, but with dhclient not getting router via DHCP.
See below the situation.

Code: 
[root@freebsd-13_1 ~]# cat /etc/rc.conf
hostname="freebsd-13_1"

ifconfig_em1="SYNCDHCP fib 1"
ifconfig_em0="SYNCDHCP fib 0"

sshd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
[root@freebsd-13_1 ~]#
[root@freebsd-13_1 ~]# cat /etc/rc.conf.d/dhclient
unset dhclient_fib
[ "$ifn" = "em1" ] && dhclient_fib=1
[root@freebsd-13_1 ~]#
[root@freebsd-13_1 ~]#
[root@freebsd-13_1 ~]# setfib 0 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.2.1.1           UGS         em0
10.2.1.0/24        link#1             U           em0
10.2.1.70          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
[root@freebsd-13_1 ~]# setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
10.13.13.0/24      link#2             U           em1
10.13.13.88        link#2             UHS         lo0
127.0.0.1          link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
ff02::/16                         ::1                           UGRS        lo0
[root@freebsd-13_1 ~]# grep router /var/db/dhclient.leases.em1
[root@freebsd-13_1 ~]#
[root@freebsd-13_1 ~]#
[root@freebsd-13_1 ~]# grep router /var/db/dhclient.leases.em0
  option routers 10.2.1.1;
  option routers 10.2.1.1;
[root@freebsd-13_1 ~]#
 
my guess is dhclient wont get/set default route if already exists
so the config was to force dhclient to run in fib 1 which has no default route
i did not test it by rebooting but /etc/rc.d/dhclient start wlan0 works for me (it sets a default route)
if i remove the conf/hack it wont set a default route
 
covacat said:
my guess is dhclient wont get/set default route if already exists
so the config was to force dhclient to run in fib 1 which has no default route
i did not test it by rebooting but /etc/rc.d/dhclient start wlan0 works for me (it sets a default route)
if i remove the conf/hack it wont set a default route
Click to expand...
Thanks, I'll try that when I get home, but won't hold my breath as I already tried the following without success (writing on phone from memory) :
setfib 1 dhclient - b em1
 
rebooted and works
Code: 
[root@hp430 ~]# grep -v ^# /etc/sysctl.conf

net.fibs=2
[root@hp430 ~]# grep ifconfig /etc/rc.conf
ifconfig_re0="DHCP"
ifconfig_wlan0="ssid shinbet2 syncdhcp fib 1"
[root@hp430 ~]# cat /etc/rc.conf.d/dhclient
unset dhclient_fib
[ "$ifn" = "wlan0" ] && dhclient_fib=1
[root@hp430 ~]# netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.1.1.1           UGS         re0
10.1.1.0/24        link#1             U           re0
10.1.1.181         link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
[root@hp430 ~]# setfib 1 netstat -rn -f inet
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.77.1       UGS       wlan0
127.0.0.1          link#2             UHS         lo0
192.168.77.0/24    link#3             U         wlan0
192.168.77.178     link#3             UHS         lo0
 
covacat said:
rebooted and works
Code: 
[root@hp430 ~]# grep -v ^# /etc/sysctl.conf

net.fibs=2
[root@hp430 ~]# grep ifconfig /etc/rc.conf
ifconfig_re0="DHCP"
ifconfig_wlan0="ssid shinbet2 syncdhcp fib 1"
[root@hp430 ~]# cat /etc/rc.conf.d/dhclient
unset dhclient_fib
[ "$ifn" = "wlan0" ] && dhclient_fib=1
[root@hp430 ~]# netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.1.1.1           UGS         re0
10.1.1.0/24        link#1             U           re0
10.1.1.181         link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
[root@hp430 ~]# setfib 1 netstat -rn -f inet
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.77.1       UGS       wlan0
127.0.0.1          link#2             UHS         lo0
192.168.77.0/24    link#3             U         wlan0
192.168.77.178     link#3             UHS         lo0
Click to expand...
Thanks, but no luck for me. I mimicked your setup, but same results.
I'll open a bug report I guess, something's rotten in the state of Denmark..
 
Code: 
exec 2>>/tmp/some.log
set -x
...
ARP=/usr/sbin/arp
add this to the top of /sbin/dhclient-script just before ARP=...
then investigate /tmp/some.log
 
covacat said:
Code: 
exec 2>>/tmp/some.log
set -x
...
ARP=/usr/sbin/arp
add this to the top of /sbin/dhclient-script just before ARP=...
then investigate /tmp/some.log
Click to expand...
Thanks. Here's the output from /tmp/some.log:
pastebin.com

[root@freebsd-13_1 ~]# cat /tmp/some.log + ARP=/usr/sbin/arp+ HOSTNAME=/bin/ - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com

You'll see extra stuff in there as I have hacked an ugly workaround in /etc/dhclient-exit-hooks - luckily in this very case my DHCP server is also my router.
This works for me currently, but if you also agree I'm tempted to open a bug report against setfib. :/

Code: 
#!/usr/bin/env bash

export PATH=$PATH:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

echo "=========="
printenv

if [ "$interface" = em0 ];
        then
                echo "The interface is em0, skipping";
        else
    case "$reason" in BOUND|RENEW|REBIND|REBOOT)
# grab the interface number
ifnumber=$(echo "$interface" | grep -Eo '[0-9]{1,4}')

# let's hope the DHCP server is also the router

setfib "$ifnumber" route add default "$new_dhcp_server_identifier"
    ;;
    esac
fi
 
try this while doing dhclient em1
Code: 
# tcpdump -i wlan0 -nl -vvv -s2222 udp and \(src port 67 or src port 68\)
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 2222 bytes
22:43:28.579914 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 50:2b:73:c9:00:2d, length 300, xid 0x2c7fb141, Flags [none] (0x0000)
      Client-Ethernet-Address 50:2b:73:c9:00:2d
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Discover
        Client-ID Option 61, length 7: ether 50:2b:73:c9:00:2d
        Hostname Option 12, length 5: "hp430"
        Parameter-Request Option 55, length 10:
          Subnet-Mask, BR, Time-Zone, Classless-Static-Route
          Default-Gateway, Domain-Name, Domain-Name-Server, Hostname
          Option 119, MTU
        END Option 255, length 0
        PAD Option 0, length 0, occurs 28
22:43:28.583138 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    192.168.77.1.67 > 192.168.77.178.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2c7fb141, Flags [none] (0x0000)
      Your-IP 192.168.77.178
      Server-IP 10.1.1.1
      Client-Ethernet-Address 50:2b:73:c9:00:2d
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Offer
        Server-ID Option 54, length 4: 192.168.77.1
        Lease-Time Option 51, length 4: 5781
        Subnet-Mask Option 1, length 4: 255.255.255.0
        Default-Gateway Option 3, length 4: 192.168.77.1
        Domain-Name Option 15, length 11: "guest.local"
        Domain-Name-Server Option 6, length 4: 10.1.1.1
        END Option 255, length 0
        PAD Option 0, length 0, occurs 13
22:43:30.629890 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 50:2b:73:c9:00:2d, length 300, xid 0x2c7fb141, Flags [none] (0x0000)
      Client-Ethernet-Address 50:2b:73:c9:00:2d
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Request
        Server-ID Option 54, length 4: 192.168.77.1
        Requested-IP Option 50, length 4: 192.168.77.178
        Client-ID Option 61, length 7: ether 50:2b:73:c9:00:2d
        Hostname Option 12, length 5: "hp430"
        Parameter-Request Option 55, length 10:
          Subnet-Mask, BR, Time-Zone, Classless-Static-Route
          Default-Gateway, Domain-Name, Domain-Name-Server, Hostname
          Option 119, MTU
        END Option 255, length 0
        PAD Option 0, length 0, occurs 16
22:43:30.633223 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    192.168.77.1.67 > 192.168.77.178.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2c7fb141, Flags [none] (0x0000)
      Your-IP 192.168.77.178
      Server-IP 10.1.1.1
      Client-Ethernet-Address 50:2b:73:c9:00:2d
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 192.168.77.1
        Lease-Time Option 51, length 4: 5779
        Subnet-Mask Option 1, length 4: 255.255.255.0
        Default-Gateway Option 3, length 4: 192.168.77.1
        Domain-Name Option 15, length 11: "guest.local"
        Domain-Name-Server Option 6, length 4: 10.1.1.1
        END Option 255, length 0
        PAD Option 0, length 0, occurs 13
 
covacat Thanks for that, it put me in the right direction. According to the tcpdump I am not getting back a default gateway.

What threw me off was the fact using the same networks didn't present the same problem on Linux.. So I went back and checked Debian in the same 2 networks, indeed no default gateway either in tcpdump/dhcpdump. I can only conclude Linux "cheats" and uses the "server identifier" instead. So the problem is on the router.

Thanks for all your help and patience.
 
You must log in or register to reply here.
Back
Top