Sockets Stuck in FIN_WAIT_1 (and other)

C

ckozler

Hi,

I have a few issues here that seem to boil down to something somewhere in the networking stack. I haven't 100% ruled out the network card potentially being an issue but note that I have them in an active/passive bonded configuration.

First, I use this server as a data backup and warehouse storage server so as you could imagine I have a lot of different hosts pushing data to this server all the time through rdiff-backup, rsync and SSH, and SCP. For some reason or another these connections will get stuck in some middle-way state and will never release. The processes will hang and hang and eventually eat-up the memory. Please note, I did not check the output of netstat during this phase (my mistake).

Second, I was doing a portsnap fetch update and the update was fairly large and it took me a number of restarts of the command to get it to finally complete. I finally realized it was getting hung up on the network.

Now I have all of this:

Code: 
tcp4       0  33144 10.99.90.80.30850      update5.freebsd..http  FIN_WAIT_1
tcp4       0  33144 10.99.90.80.44903      update5.freebsd..http  FIN_WAIT_1
tcp4       0  33166 10.99.90.80.40569      update5.freebsd..http  FIN_WAIT_1
tcp4       0  33145 10.99.90.80.40344      update5.freebsd..http  FIN_WAIT_1
tcp4       0  41357 10.99.90.80.33943      update5.freebsd..http  FIN_WAIT_1

What is strange to me is, minus the last one, they almost all got hung up at the same time Send_Q size. Has anyone seen this? As of now my servers arent backing up to this machine anymore because in reality the backups were technically constantly failing due to not being able to establish the proper connection (or stepping on previous jobs that were hung).

Thank you!
 
Yes, there is a hardware Juniper router and firewall in between, but this is the only server that experiences this issue. I have implemented the following and so far it looks to be better

Originally:

Code: 
$ sysctl -a | grep -i keep     
net.inet.ip.keepfaith: 0
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.keepinit: 75000
net.inet.tcp.always_keepalive: 1
net.inet6.ip6.keepfaith: 0

Now:

Code: 
$ sysctl -a | grep -i keep
net.inet.ip.keepfaith: 0
net.inet.tcp.keepidle: 1000
net.inet.tcp.keepintvl: 5000
net.inet.tcp.keepinit: 75000
net.inet.tcp.always_keepalive: 1
net.inet6.ip6.keepfaith: 0

Also, another issue I was having with the hanging rsync and SSH connections does not seem to be an issue anymore though this could have been resolved by upgrading my SSH installation and using openssh-portable as well.
 
You must log in or register to reply here.
Back
Top