Netstat hangs session

S

scryptkiddy

Today I was getting indications of an issue with a FreeBSD 8.3-RELEASE 64bit server when I noticed hundreds of sendmail processes in the process table. Digging deeper, I also noticed hundreds of netstat commands in the process table that came from the /etc/periodic/daily/420.status-network script which daily executed the netstat -i && rc=0 || rc=3;; command.

So I tried to execute different netstat commands to see if netstat was the problem using different opt switches, and every time it would hang and never return the shell prompt. So I would have to reconnect eventually.

Thoughts?

SK
 
Use the -n option to disable name lookups ;)

From netstat(1):
Code: 
     [file]-n[/file]    Show network addresses and ports as numbers.  Normally netstat
           attempts to resolve addresses and ports, and display them symboli-
           cally.
 
The -n option also causes it to hang. I did a ktrace on the netstat command: ktrace -t + -f netstat.out netstat -i and looking at netstat.out:

The last entry is a KLDSYM_LOOKUP:
Code: 
 ...
 97170 netstat  NAMI  F64  "/dev/kmem"
 97170 netstat  RET   F64  open 4
 97170 netstat  CALL  F64  fcntl(0x4,F_SETFD,FD_CLOEXEC)
 97170 netstat  RET   F64  fcntl 0
 97170 netstat  CALL  F64  getgid
 97170 netstat  RET   F64  getgid 0
 97170 netstat  CALL  F64  setgid(0)
 97170 netstat  RET   F64  setgid 0
 97170 netstat  CALL  F64  kldsym(0,KLDSYM_LOOKUP,0x7fffffffde30)

So the CALL is hanging. Suggestions?
 
I have no idea ;) On a FreeBSD 9.1p6 systems it continues successfully from there...
Code: 
...
 19664 netstat  NAMI  "/dev/kmem"
 19664 netstat  RET   open 4
 19664 netstat  CALL  fcntl(0x4,F_SETFD,FD_CLOEXEC)
 19664 netstat  RET   fcntl 0
 19664 netstat  CALL  getgid
 19664 netstat  RET   getgid 0
 19664 netstat  CALL  setgid(0)
 19664 netstat  RET   setgid 0
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffce20)
 19664 netstat  RET   kldsym 0
 19664 netstat  CALL  getpid
 19664 netstat  RET   getpid 19664/0x4cd0
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym -1 errno 2 No such file or directory
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym -1 errno 2 No such file or directory
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym -1 errno 2 No such file or directory
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym 0
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym 0
 19664 netstat  CALL  kldsym(0,KLDSYM_LOOKUP,0x7fffffffbaa0)
 19664 netstat  RET   kldsym 0
 19664 netstat  CALL  stat(0x7fffffffb4b0,0x7fffffffb430)
 19664 netstat  NAMI  "/usr/share/nls/C/libc.cat"
 19664 netstat  RET   stat -1 errno 2 No such file or directory

Considering the following information I am afraid nobody will be motivated to do something about it.

From http://www.freebsd.org/releases/
Code: 
Prior Releases Which Have Reached End-Of-Life

Complete historical information about the release date, the classification type, and 
the effective End-Of-Life (EOL) for these releases may be found on the
 Unsupported Releases section of the FreeBSD Security Information page.

    9.0 (January 2012) Announcement: Release Notes: Installation Instructions: Hardware Notes: Readme: Errata
    [color=blue]8.3 (April 2012)[/color] Announcement: Release Notes: Installation Instructions: Hardware Notes: Readme: Errata
 
Thanks for the replies, @J65nko. I did try the same command on three other servers running the same 8.3 version, they all run just fine. It seems system specific. I don't have an issue rebooting the server, but I'd rather not if the cause is something easy to fix.
 
Last edited by a moderator:
Does the output of top(1) about memory/swap usage give any clue?

It could be that all those hundreds of netstat and are depriving the kernel from the minimal memory it needs to function. You could try to kill all those netstat processes with pkill(1) and see whether that improves things.
 
Yeah, many many netstats. I killed all the netstats processes and it still hangs. I'm assuming at this point the reboot is needed. I just don't like to reboot FreeBSD, that's usually a Windows fix ;)

SK
 
You must log in or register to reply here.
Back
Top