TCP packets Drop !!

S

shahzaib

Hi,

We've encountered tons of packets being dropped using command netstat -s on FreeBSD-10.1. The one whose timeout value keeps on increasing is 'rexmit timeout' . Here is the list of timeout / dropped packets.

http://prntscr.com/aod4f2

The server is on 4x1Gbps LACP port . We've also tried disabling NICs feature TSO but in vain :

http://prntscr.com/aod526

Please let me know if you need any further info to help resolving this issue ?

Thanks in advance !!
 
Last edited by a moderator:
We have no way to tell if this is really a problem because you have not provided all of the information required. A certain percentage of retransmissions and connection drops should be considered normal. Since you only provided a small part of the statistics, and no information on your application, we can't see whether this drop rate represents .1% or 90% of connections.

It might also be useful to run a netstat -i and see if there are any packets with errors at the ethernet level. Depending on what you have for switches, you may be able to get more information from them about per-port issues. Unfortunately, it is not possible to get per-interface TCP statistics using netstat: that would be great for seeing if the drops are all on one port (e.g. incoming Internet connections).

If you still need us to look at this information, maybe you should redirect the statistics output to a file (i.e. netstat -s >stats-file) and paste the whole thing in here inside of a [ CODE ] block.
 
I am assuming this is Internet traffic, rather than local to your LAN. Also, it looks like you are sending way more data out than you are receiving (110TiB sent, 612GiB received).

About 6 TCP packets per million are being dropped due to bad checksums (252397/40077338123). That is about twice as high as I am experiencing on Internet traffic, but probably nothing to worry about.

About 1.5% of the bytes you are sending get retransmitted, and that is about a quarter of what I am experiencing.

About 3% (776813/26329090) of your TCP connections are dropped due to retransmit timeouts. That is about 30 times as high as I am experiencing. This is most likely due to persistent congestion, or network outages, somewhere between your server and the client machine. It would be really useful to know if this is happening more to some clients than others (e.g. is your server logging this someplace, and if so, are there any patterns in the client IP addresses). If your connection to your upstream provider is maxxed out then that could be a big part of the problem, but otherwise you are going to have a tough time improving this. Is your upstream router dropping lots of outgoing packets due to congestion? That could be a big indicator that you need more bandwidth.
 
Pardon me for missing some details. Yes its internet traffic, we serve vod .mp4 content over nginx web-server to users that is the reason outgoing bandwidth is so high. Usual realtime traffic is 800Mbps while the peak is around 3Gbps.

Regarding bandwidth congestion to upstream router, we don't believe that could be the case because we've 8Gbps port commitment with Data-Center while burst is 10Gbps and peak traffic has always touched not more than 8Gbps but for upstream provider (I believe it is ISP Transit Provider) i'll have to talk to Data-Center guys.

>> It would be really useful to know if this is happening more to some clients than others (e.g. is your server logging this someplace, and if so, are there any patterns in the client IP addresses)
Would nginx logs help here ?

Thanks a lot for your detailed explanation to help with our case. :)
 
If this is Internet traffic where your packets are lost, I can mention at least 2 reasons I personally encountered: a faulty DSL modem or a faulty DSL transmitter of the physical line provider.
Replacing the modem with the identical but working one fixed the 1st case, and in the 2nd case the provider (Bell Canada) sent a tech who replaced the transmitters in both demarcation box and concentrator.
Oh, and the 3d one: wrong MTU. If your MTU is too large the packets may be dropped when being fragmented.
 
You must log in or register to reply here.
Back
Top