sockstat showing question marks when trying to close socket

[ecd]

I'm trying to debug an issue where sockets aren't being properly closed in a Java application. It works under Linux, but with FreeBSD a socket appears to be left open though with no associated file descriptor (according to sockstat)

While the server on port 9999 is running, here's some output from sockstat and netstat:

[ecd@qk ~]$ sockstat -4 | grep 9999
ecd      java       48847 17 tcp4   127.0.0.1:9999        *:*
[ecd@qk ~]$ netstat -an | grep 9999
tcp4       0      0 127.0.0.1.9999         *.*                    LISTEN

It looks correct and as expected. Now, when the app closes its server socket, the following system call is seen from truss:

48847: dup2(0x18,0x11,0x8028769a8,0xc,0x80469e000,0x0) = 17 (0x11)

The 2nd argument is the same file descriptor listening on 9999 (fd 17), so my understanding is that it should be close()'d then. (BTW, Why does truss show all those extra arguments? Shouldn't there only be 2?)
Following this, the output from sockstat/netstat is:

[ecd@qk ~]$ sockstat -4 | grep 9999
?        ?          ?     ?  tcp4   127.0.0.1:9999        *:*
[ecd@qk ~]$ netstat -an | grep 9999
tcp4       0      0 127.0.0.1.9999         *.*                    LISTEN

and although the app thinks it's closed the socket, it is still lingering around.

Does anyone know what's happening to the socket here? Why is sockstat displaying question marks?

(For context, here is the minimal app I've been testing with: https://gist.github.com/ericdahl/0dc63c44cd05a6f5bd9a)

 

[worldi]

Re: sockstat showing question marks when trying to close soc

At a glance this looks like a SO_REUSEADDR problem. However, in this case netstat would (or should, in case it is a bug) report TIME_WAIT instead of LISTEN...

About the truss issue:
truss has a hardcoded list of syscall metadata which is incomplete. Any syscall not in the list will be displayed with six arguments ( #define MAXARGS 6). Where the surplus ones are taken from depends on the ABI.

About the question marks displayed by sockstat:
This is most probably because the process the socket belonged to has terminated (i.e. there's no useful information available). The kernel, however, has to keep the socket around for a while because closing a TCP connection requires a 4-way handshake and the process terminated after the first round.

 

[ecd]

Re: sockstat showing question marks when trying to close soc

Good to know about truss.

The application is setting SO_REUSEADDR.

When I let the app close its socket (via dup2) and then let it sit there idling (debug session open), the sockstat/netstat output remains unchanged after 5+ minutes. My understanding is that TIME_WAIT is ~1 minute.

Interestingly, if I actually send in a client request/connection while the server is started, and then stop the server, it appears to close the socket. I can successfully use nc -lv 127.0.0.1 9999 but if a client does not connect when the server is started, I'll get a

[ecd@qk ~]$ nc -lv 127.0.0.1 9999
nc: Address already in use

That doesn't seem like it should happen. I guess I'll have to dig some more. I'd like to at least try to reproduce it in a simpler C program that just uses the system calls.