'named: not enough free resources' gets box offline

It happend couple times when the production box went offline but still running.
The Reset helped only.
Later on I found in:
/var/log/messages
a lot of strings matching exactly at the time when box went offline:
Code:
Jan  8 20:00:00 freebsdbox named[605]: client 212.0.221.131#63394: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.5.151#46650: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 4.68.25.4#38728: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.0.219.67#39168: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.101.62#21972: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.86.243#37008: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.93.158.242#21273: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.101.62#51570: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 212.0.209.131#24539: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 67.202.21.1#13739: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 85.37.17.17#52719: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 174.129.182.213#28260: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.5.151#34352: error sending response: not enough free resources
Jan  8 20:00:01 freebsdbox named[605]: client 193.226.6.233#54745: error sending response: not enough free resources

Just in case may be interested in current output for network_card is:
# # ifconfig -a
Code:
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1d:92:61:54:95
        inet xx.xx.69.211 netmask 0xffffffe0 broadcast xx.xx.69.223
        inet xx.xx.215.24 netmask 0xffffffff broadcast xx.xx.215.24
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet6 ::1 prefixlen 128 
        inet 127.0.0.1 netmask 0xff000000

Any ideas are very welcome on where to dig so to solve the issue ASAP, please.
 
Here is:
# # netstat -m
Code:
66/1854/1920 mbufs in use (current/cache/total)
64/466/530/65536 mbuf clusters in use (current/cache/total/max)
64/448 mbuf+clusters out of packet secondary zone in use (current/cache)
0/199/199/32768 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
144K/2191K/2336K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
 
Check you firewall (if you have one running). Also make sure you don't have any network problems like a dodgy card or cable.
 
Ok, regarding making sure about any network problems like a dodgy card or cable -- it's hard to go for it, because the box inside the DataCenter.
So, I've just disabled IPFW to see if it could cause the issue.
And for your review, here is the:
# # ipfw list
Code:
[color="Red"]00100 allow ip from any to any via lo0[/color]
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 reset tcp from 173.45.76.212 to me dst-port 80
00500 reset tcp from 202.222.31.109 to me dst-port 80
00600 reset tcp from 74.143.62.214 to me dst-port 80
00700 reset tcp from 87.106.252.228 to me dst-port 80
00800 reset tcp from 115.68.53.195 to me dst-port 80
00900 reset tcp from 202.73.57.25 to me dst-port 80
01000 reset tcp from 195.154.126.140 to me dst-port 80
01100 allow tcp from any to me dst-port 80
01200 reset tcp from 202.73.57.25 to me dst-port 22
01300 reset tcp from 115.68.53.195 to me dst-port 22
01400 reset tcp from 202.154.57.33 to me dst-port 22
01500 reset tcp from 87.106.252.228 to me dst-port 22
01600 reset tcp from 74.143.62.214 to me dst-port 22
01700 reset tcp from 202.222.31.109 to me dst-port 22
01800 reset tcp from 173.45.76.212 to me dst-port 22
01900 reset tcp from 187.5.128.234 to me dst-port 22
02000 reset tcp from 124.42.41.162 to me dst-port 22
02100 reset tcp from 195.154.126.140 to me dst-port 22
02200 allow tcp from any to me dst-port 22
02300 reset tcp from 173.45.76.212 to me dst-port 21
02400 reset tcp from 202.222.31.109 to me dst-port 21
02500 reset tcp from 74.143.62.214 to me dst-port 21
02600 reset tcp from 87.106.252.228 to me dst-port 21
02700 reset tcp from 115.68.53.195 to me dst-port 21
02800 reset tcp from 202.73.57.25 to me dst-port 21
02900 reset tcp from 130.58.84.55 to me dst-port 21
03000 reset tcp from 75.151.84.17 to me dst-port 21
03100 reset tcp from 87.237.208.232 to me dst-port 21
03200 reset tcp from 85.17.45.56 to me dst-port 21
03300 reset tcp from 66.199.248.58 to me dst-port 21
03400 reset tcp from 195.154.126.140 to me dst-port 21
03500 allow tcp from any to me dst-port 21
65535 allow ip from any to any

*The strange thing for me is the very 1st line ouput stating
that Netword Card Interface is 'lo0', while real is 're0'
 
joint said:
The strange thing for me is the very 1st line ouput stating
that Netword Card Interface is 'lo0', while real is 're0'

lo0 is the loopback adapter (where 127.0.0.1 is bound to).
 
Ok, Thank you for your explanation regarding 'lo0'
and mega-appreciate you for the things pointed to me.
Apart from them is it any extra ideas of what may cause the issue?
In the mean time I'm runnig with IPFW disabled.
Here is the my system details:
# # uname -a
Code:
7.0-RELEASE FreeBSD 7.0-RELEASE #0: Thu Feb 28 05:32:36 UTC 2008     [email]root@orion.ispsystem.net[/email]:/usr/obj/usr/src/sys/ISPSYSTEM  amd64
 
joint said:
Apart from them is it any extra ideas of what may cause the issue?
In the mean time I'm runnig with IPFW disabled.
Out of ideas but I am wondering if it works with IPFW disabled. If that's the case then the problem is with the firewall and/or it's settings.

Here is the my system details:
# uname -a
7.0-RELEASE FreeBSD 7.0-RELEASE #0: Thu Feb 28 05:32:36 UTC 2008 root@orion.ispsystem.net:/usr/obj/usr/src/sys/ISPSYSTEM amd64
I do hope named isn't accessible from the outside. I strongly suggest updating.


http://security.freebsd.org/advisories/FreeBSD-SA-08:06.bind.asc
http://security.freebsd.org/advisories/FreeBSD-SA-09:04.bind.asc
http://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc
 
SirDice said:

Ok, I've never done any patching before and would like to ask you let me know specifics about it.
1. There are /usr/src but it is empty...probably I have to do something for it?
2. Is it correct to do all 3 patches one by another rebooting the box everytime starting from the top link you've provided?
3. The current kernel I've got is not default GENERIC,
it is custom and has been created when I've installed ISPManager Server Control Panel http://ispsystem.com/en/
and not sure if I'll be able to start up the box once things will be patched.
Your thoughts & instructions, please?
 
1) http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/updating-upgrading.html
The source update mentions cvsup, there's no need to install it. Since 6.4 the base OS has csup which is similar in features and configuration.

2) You can do all patches at once and build just once.

3) Shouldn't be a problem but it does mean you can't use freebsd-update. If you want to use that you'll need to temporarily switch back to GENERIC.

As for updating, that last bind problem isn't fixed in 7.0 because 7.0 isn't supported anymore. If you update I suggest updating to 7.2-RELEASE-p6. This should be a relatively easy update. No need to rebuild any ports AFAIK. Do check /usr/src/UPDATING before starting.
 
Thank you very much!
I'll try to do all that in the night time.
As soon as I'll get things done I'll be back with the feedback.
Once again I appreciate you for your time & effort.
 
Here is the Feedback:
I Found that command 'top' has strange output now giving this line:
Code:
[I]top: sysctl(vfs.bufspace...) expected 8, got 4[/I]
I've got this right after I had made:
Code:
# csup -g -L2 /usr/local/etc/stable-supfile
# cd /usr/src && make buildworld && make installworld
# reboot
# mergemaster -Ui
# reboot

I assume that I have to change some values in sysctl.conf,
but I'm not sure which one exactly needs to be changed.
The Amount of RAM installed is 4Gb.
Any ideas, please?
 
You mean you updated the world, but not the kernel? That's a recipe for disaster, and it is why 'top' is screwed: the kernel doesn't understand userland anymore. What made you take the steps listed above, instead of the proper procedure listed in /usr/src/Makefile?
 
Well, looks like now I'm screwed up for sure.
Is it any short-hand out_of_current_disaster recipe you may suggest to me?
Bear in mind, that I have not GENERIC kernel running.
And when I tried 'freebsd-update fetch' 'freebsd-update install' and then reboot - I lost box available by SSH and then had to wait for hours when Data-Center support getting me KVM_over_IP console connected and only 'freebsd-update rollback' could get that box back online.
Is it any specific suggestions on this, please?
 
Either use freebsd-update, or update from sources, and if you decide to update world/kernel from sources, use the correct and full procedure.

You can use a non-GENERIC kernel with freebsd-update (by letting freebsd-update update the sources it uses to build its binaries, and building a custom kernel for your system, while keeping the GENERIC kernel in /boot/GENERIC) - but do not mix freebsd-update and make world! Use one or the other, and stick with it.

I have no idea what your system is running right now, but I suggest you first run freebsd-update and reinstall and reboot the GENERIC kernel installed by freebsd-update (make sure 'src' is updated as well), and then try building a custom kernel again (keeping a copy of the generic kernel in /boot/GENERIC).

The Handbook has more on that subject.
 
Ok, thank you guys for making things brighter now.
So, let me know, please, if I've got you the right way describing steps below with commands:
1. To update /usr/ports;

2. To update the 'src' with:
Code:
   # csup -g -L2 /usr/local/etc/stable-supfile

3. To update FreeBSD -- I've got Release 7.0 for now:
Code:
   # freebsd-update fetch
   # freebsd-update install

4. Build a custom kernel with needed OPTIONS enabled:
Code:
   # cd /usr/src/sys/`uname -p`/conf
   # cp GENERIC ISPSYSTEM         --The ISPSYSTEM is the name of my current custom kernel + with 3 OPTIONS added to GENERIC 
   # vi ISPSYSTEM
... add those 3 OPTIONS that needed over again and after that go:
   # cd /usr/src          
 [color="DarkOrange"]  # make buildworld  --I'm NOT sure if should I do this command??[/color]
   # make buildkernel KERNCONF=ISPSYSTEM   
   # make installkernel KERNCONF=ISPSYSTEM

5. Reboot:
Code:
   # shutdown -r now

Are those steps above sound correct to follow?
Please, correct me if I'm wrong somewhere.
Thank you!
 
dennylin93 said:
If you have a CD/DVD with FreeBSD, you can use it to get a GENERIC kernel as well.
That is the case, that I've got this box remotely far away installed in Data_Center.
Thanks for paying attention to my post as well as advice;)
 
Back
Top