Manual rc.conf routes are eventually removed automatically.

[urbanriot]

I have a FreeBSD proxy / router configured at a location with multiple internet connections and I've configured multiple routes in rc.conf for various CIDR ranges to pass through each of the various internet connections. However after an undetermined amount of time, all of the routes are missing and I have to either manually re-add them or restart the system to re-establish them.

Is there any way I can prevent this from happening? I have other FreeBSD routers in other locations, albeit with older versions of FreeBSD, that are configured exactly the same yet don't exhibit this automated route deletion behaviour.

 

[trh411]

Re: Manual rc.conf routes are eventually removed automatical

Any clues in the log file? What is the output of uname -a for the problem router? It would be good to see your /etc/rc.conf file and your routing table when all routes are present.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

Any clues in the log file?

No, it's basically clean of anything network related. I'm now keeping the ASDM of a Cisco firewall that receives one of these routes on screen, watching the bandwidth for the day(s) until I see a change indicating the issue and I'll see if I can correlate that to... anything.

What is the output of uname-a for the problem router?

FreeBSD internet 9.1-RELEASE-p7 FreeBSD 9.1-RELEASE-p7 #0: Fri Oct 25 00:10:29 EDT 2013     root@internet:/usr/obj/usr/src/sys/SYSv3  i386

It would be good to see your /etc/rc.conffile and your routing table when all routes are present.

Unfortunately I can't provide that for privacy reasons as the ranges are all company specific in an indirect way. Suffice to say each route that's added is error-free and accepted at boot time, and working until they all disappear. So far the disappearing time is random but now that I'm more aware / frustrated with this issue I'm more focused on the issue so I may note a specific time.

Two of the interfaces are DHCP and the rest are static, could that be encouraging the issue? The DHCP interfaces have not changed in months so that change is not triggering the issue but I figured I'd throw that configuration difference into the mix. The static routes created in rc.conf are directed at a statically assigned interface, not a DHCP assigned interface.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

This morning I discovered the static routes were all removed and the routing table was basically defaulted.

The router passing these static routes to the internet had not gone offline so I don't know what's causing this behaviour. Is there an automated mechanism of FreeBSD that's evaluating these static routes and deleting them after a patterned time or series of events?

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

This morning I discovered the static routes were all removed and the routing table was basically defaulted.

And again this morning. So far I've established that it's a nightly occurrence.

Is there an automated mechanism of FreeBSD that's evaluating these static routes and deleting them after a patterned time or series of events?

So this is where I assume I need to focus, "Is there an automated mechanism of FreeBSD that's evaluating these static routes and deleting them" each night? I looked at the /periodic/daily scripts which are invoked each night in crontab and I see nothing that might cause this.

 

[trh411]

Re: Manual rc.conf routes are eventually removed automatical

How about a script that runs every minute (or even every few seconds) to monitor the routing table to pinpoint exactly when the routes are removed?

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

How about a script that runs every minute (or even every few seconds) to monitor the routing table to pinpoint exactly when the routes are removed?

I'm clever in a logical way but less than clever in the scripting or programming department. Happen to have any links to anyone that's crafted such a script?

 

[trh411]

Re: Manual rc.conf routes are eventually removed automatical

I like to keep things simple, so I would start with:

#!/bin/sh                                                                                                

echo "----------------------------------------------------------------"
echo "Dumping routing table on $( hostname ) on $( date )"
/usr/bin/netstat -r
echo
echo
echo
echo

exit 0

And run it every 15 minutes from cron from 00:01 to 06:00. Redirect (append) the output to a file of your choosing. This can be run as a normal user.

The user's cron entry would look something like:

*/15	0-5      *	*	*    /path/to/script.sh >>/path/to/log/file

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

Great idea, thanks! Route dumping to log file has been initiated and I'll report back after I have some data!

 

[SirDice]

Re: Manual rc.conf routes are eventually removed automatical

The router passing these static routes to the internet had not gone offline so I don't know what's causing this behaviour. Is there an automated mechanism of FreeBSD that's evaluating these static routes and deleting them after a patterned time or series of events?

Maybe the interface that's used by these routes disappears briefly? That could remove the routes.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

And run it every 15 minutes from cron from 00:01 to 06:00. Redirect (append) the output to a file of your choosing. This can be run as a normal user.

Day 1 of reporting - Routes listed at 1:30 AM EST, routed deleted by 1:40 AM EST (Ii went with 10 minute intervals rather than 15).

I'll report back for Day 2 and see if it's the same time.

The router passing these static routes to the internet had not gone offline so I don't know what's causing this behaviour. Is there an automated mechanism of FreeBSD that's evaluating these static routes and deleting them after a patterned time or series of events?

Maybe the interface that's used by these routes disappears briefly? That could remove the routes.

Interesting point. I was thinking that since this particular Cisco router is passing data that hosts a web site for 24/7 uptime, it most certainly should not be going down; however I hadn't considered that the NIC in this FreeBSD router might be dropping out for some strange reason. I suppose by tomorrow I'll have more information to determine if I should focus on a software pattern or a hardware issue.

Thanks for the interaction guys!

 

[trh411]

Re: Manual rc.conf routes are eventually removed automatical

@SirDice's suggestion is an interesting one. It certainly begs the question "what would cause the interface to (briefly) disappear"? If the Day 2 results show the same time interval for the disappearing routes, it will beg an even more intriguing question "what would cause the interface to (briefly) disappear at the same time every night"?

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

Day 2 of reporting - I rebooted the router at 6:30 PM EST yesterday and this morning at 9:30 AM the routes are still intact. Will report back when they've dropped. I suppose I should also consider if it's timed rather the same hour day as well.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

... and on Day 2, the routes were lost at some point between 8:20 PM and 8:30 PM EST, pretty much 10 hours since the last time. Will update again tomorrow.

 

[trh411]

Re: Manual rc.conf routes are eventually removed automatical

Thanks for the update. I'm really interested to see how this plays out. Was the router reboot intended as part of your problem determination or for some unrelated reason? Just curious.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

The router I'm referring to is the FreeBSD router, not the Cisco router which is never rebooted, and I rebooted it since its uptime isn't critical and it was quick and easy to recreate the routes via rc.conf and time the uptime.

 

[kpa]

Re: Manual rc.conf routes are eventually removed automatical

Any suspicious messages in dmesg(8) output? If the interface gets reset for some reason there could be UP/DOWN messages in the log.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

Any suspicious messages in dmesg(8) output? If the interface gets reset for some reason there could be UP/DOWN messages in the log.

No, unfortunately it's clean, for days since the last time it was booted.

On Day 3 I have the routes dropping around lunch today so it doesn't seem like a pattern after all. I might use an alternative NIC to diagnose this further. I'm using the second port of an Intel dual-port server NIC that uses em(4) drivers (device em # Intel PRO/1000 Gigabit Ethernet Family) and I'd expect the card to be reliable.

It will probably be a few more days before I have any updated data.

Edit: I'm also going to start reviewing the Cisco logs to see if this FreeBSD system has disconnected at any point.

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

...fast forward many months since my original posts and after reviewing Cisco logs and other connected system logs, and replacing the FreeBSD system in question, I'm still experiencing a problem where my routes are randomly rejiggered automatically while there's no log entries referencing any kind of issue.

So my question is, how do I make persistent routes really persistent, as in, they won't be automatically wiped out by the default behaviour of whatever it is that wipes them out? I understand this functionality exists for a reason but I have a need to maintain a good 50+ manual routes and automating a script is less reliable than simply disabling this functionality.

Thanks in advance!

 

[urbanriot]

Re: Manual rc.conf routes are eventually removed automatical

I hate bumping but I'm really stuck here, I have the same experience on a new system I'd swapped to try and get around this problem but with considerably less frequency. Basically if I have a pile of custom routes, they're all wiped out after a seemingly random period and I want to disable whatever aspect of FreeBSD causes that to happen.