Default route not surviving reboot and randomly disappearing

LITMK

LITMK

Hello,

I'm running FreeBSD 12.1-RELEASE-p3 and my /etc/rc.conf has my network set up like this:

Code: 
ifconfig_igb1="inet 10.0.0.20 netmask 255.255.255.0"
defaultrouter="10.0.0.1"

Lately when I reboot the machine the default route is not set and I have to add it manually with:

Code: 
route add default 10.0.0.1

Sometimes after being up for a while the default route disappears and I've confirmed this with netstat. I then have to add it manually again.

I don't know where to start diagnosing this and search engine results have turned up a bug from FreeBSD 9.0 where the default route would change to another address under high load, but that's not my problem.

I appreciate any help or suggestions on where to look or what to check.
 
LITMK said:
Sometimes after being up for a while the default route disappears and I've confirmed this with netstat.
Click to expand...
This could be the result of the interface disappearing and reappearing. When an interface is removed (for whatever reason) the accompanying routes also disappear. When the interface comes back those routes will still be gone. Check your /var/log/messages, does that provide any clues to the reason?
 
You figured it out. Thank you!

The logs show the interface going down and coming up a few times.

I don't run a huge amount of traffic on this machine. About a dozen jails and a bhyve Ubuntu game server.

Do you have any ideas as to what could be causing this problem? I really have no idea where to start looking.

It's onboard NIC, Intel server board (Intel S5520HC).
 
LITMK said:
The logs show the interface going down and coming up a few times.
...
Do you have any ideas as to what could be causing this problem?
Click to expand...
There could be several reasons. The first could be that the card is faulty and it's just limping along until it completely dies. Another reason could be a bad cable causing the connection to to be down intermittently. It could also be the port of the switch it is connected to. What exactly is being reported in /var/log/messages?
 
SirDice has said it all. If you however want to restore you route you can enter it into rc.local, where the routes are immediately loaded on start-up AND you could use Cron to periodically run the file -
Code: 
0/10 * * * * sh /etc/rc.local >/dev/null 2>=1
 
mickey said:
I'm not sure if this is related to the problem you are experiencing, but there has been an ERRATA notice regarding a problem with the igb driver, so updating your system might be something worth considering.
Click to expand...
I saw the Igb interface act up recently - altq queueing broke invariably pf. A kernel and world upgrade fixed it. We're not informed of his network interfaces and hardware.
 
With 12.0 the igb driver was integrated into the em(4) driver. It used to be a separate driver. This integration effort may have introduced bugs. I have four igb interfaces on a SuperMicro server which is running 12-STABLE since it became available. So far I haven't encountered any serious issues with it. This doesn't rule out bugs though, I may have been lucky not to have been bitten by them.

I would start by replacing the cable. If possible also try a different switch port. Those are easy to try without having to take the whole thing apart.
 
It doesn't matter if there's cable or connection to the interface when he's using static route and static ip address configuration. The routing changes are only visible on the console so he can't see them in the /var/log/messages

Most probable cause is that the driver of the interface is loaded after the route fail or he already have another default route in the table.

After the reboot check the console output for:

Code: 
route: writing to routing socket: Network is unreachable
add net  default: gateway 10.0.0.1 fib 0: Network is unreachable

If there's no connection to the interface (aka disconnected cable) but the driver is loaded before the route it will get the route with interface down.

Edit:
If you have another default route on igb0 interface configured by DHCP will cause the same issue. As you can't have 2 default gw
you will see the error
add net default: gateway 10.0.0.1 fib 0: route already in table
 
VladiBG said:
The routing changes are only visible on the console so he can't see them in the /var/log/messages
Click to expand...
In that case I would recommend enabling this in /etc/syslog.conf:
Code: 
# uncomment this to log all writes to /dev/console to /var/log/console.log
# touch /var/log/console.log and chmod it to mode 600 before it will work
console.info                                   /var/log/console.log
This will make sure the console messages are logged, making it a little easier to diagnose after the fact.
 
I installed p7 and restarted. Same thing. Here is the message:

Code: 
igb1: link state changed to UP
ums0 on uhub1
ums0: <Mouse Interface> on usbus2
ums0: 3 buttons and [Z] coordinates ID=0
uhid0 on uhub4
uhid0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 2> on usbus5
device_attach: ioapic1 attach returned 6
bridge0: Ethernet address: 02:92:6d:7b:c6:00
bridge0: changing name to 'vm-public'
igb1: link state changed to DOWN
vm-public: link state changed to UP
igb1: promiscuous mode enabled
igb1: link state changed to UP

In the meantime I've enabled logging of console messages. And I'll change the cable and port to check that.

When I first got this board igb0 was doing this up/down dance so I switched over to igb1. I think Dice might be right about the card being on it's last legs. Maybe time to get a 10G! 😍

Thanks to everyone who has been helping! I'll let you know what I find. Very much appreciated.
 
You must log in or register to reply here.
Back
Top