So problem is that several FreeBSD 12.1-RELEASE-p9 machines with interfaces em0, em1, tun0 from a while start giving an outbound connectivity errors.
It goes like when trying to connect external IP with SSH, FTP, NFS or HTTPS - 4 connections out of 10 are failed: connection timeout.
Once connection is established all is working fine. For example
I am not quite sure where to look at it. I did try:
I have a few theories:
It goes like when trying to connect external IP with SSH, FTP, NFS or HTTPS - 4 connections out of 10 are failed: connection timeout.
Once connection is established all is working fine. For example
poundiere is building 1000 ports on one of the servers and it is getting like 20 failed build due to a problem connecting with GitHub or others.I am not quite sure where to look at it. I did try:
- /var/log/messages looks clean and healthy.
- /etc/pf.conf is set up and I thought it might be a problem there but config was unchanged for years now,
- Only system updates via
freebsd-update fetch install pflog0does not show anything that gets blocked.- I did try net/mtr-nox11 to the various IP addresses and there is no packet drop. Interesting is sometimes
mtrgive me protocol error when adding -T and -u options. 2 in 10. tracerouteto a number of IP addresses works without any drops.telnetto IP's on different ports is random: >4 on 10 fail to different IP's.ncto IP's on different ports is random: >4 on 10 fail to different IP's.wgetfrom jails fail like 50% to the same IP ( not domain names ).- DNS is working fine.
I have a few theories:
- a firewall above servers is just messing around
- a router above cannot pass out packages
pfctlis doing too much out filtering ( nothing in logs orpflog) - plan to disable it.- operating system get delays somewhere
- a network interface is faulty ( but at the same time on multiple servers ? )
Bash:
#netstat -s
tcp:
15022 packets sent
9415 data packets (1482010 bytes)
1 data packet (768 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
1454 ack-only packets (28 delayed)
0 URG only packets
0 window probe packets
0 window update packets
4236 control packets
9907 packets received
8052 acks (for 1482472 bytes)
7 duplicate acks
0 acks for unsent data
2623 packets (229795 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
2 packets received after close
4 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
3715 connection requests
7 connection accepts
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
526 connections established (including accepts)
517 times used RTT from hostcache
517 times used RTT variance from hostcache
0 times used slow-start threshold from hostcache
3679 connections closed (including 2 drops)
10 connections updated cached RTT on close
10 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
3030 embryonic connections dropped
7351 segments updated rtt (of 10530 attempts)
242 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
2 keepalive timeouts
0 keepalive probes sent
2 connections dropped by keepalive
147 correct ACK header predictions
1293 correct data packet header predictions
8 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
7 completed
0 bucket overflow
0 cache overflow
1 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
8 cookies sent
0 cookies received
4 hostcache entries added
0 bucket overflow
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
0 packets with matching signature received
0 packets with bad signature received
0 times failed to make signature due to no SA
0 times unexpected signature received
0 times no signature provided by segment
0 Path MTU discovery black hole detection activations
0 Path MTU discovery black hole detection min MSS activations
0 Path MTU discovery black hole detection failures
TCP connection count by state:
0 connections in CLOSED state
27 connections in LISTEN state
0 connections in SYN_SENT state
0 connections in SYN_RCVD state
5 connections in ESTABLISHED state
0 connections in CLOSE_WAIT state
0 connections in FIN_WAIT_1 state
0 connections in CLOSING state
0 connections in LAST_ACK state
0 connections in FIN_WAIT_2 state
57 connections in TIME_WAIT state
udp:
1800 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
61 dropped due to no socket
6 broadcast/multicast datagrams undelivered
0 dropped due to full socket buffers
0 not for hashed pcb
1733 delivered
1862 datagrams output
0 times multicast source filter matched
sctp:
0 input packets
0 datagrams
0 packets that had data
0 input SACK chunks
0 input DATA chunks
0 duplicate DATA chunks
0 input HB chunks
0 HB-ACK chunks
0 input ECNE chunks
0 input AUTH chunks
0 chunks missing AUTH
0 invalid HMAC ids received
0 invalid secret ids received
0 auth failed
0 fast path receives all one chunk
0 fast path multi-part data
0 output packets
0 output SACKs
0 output DATA chunks
0 retransmitted DATA chunks
0 fast retransmitted DATA chunks
0 FR's that happened more than once to same chunk
0 output HB chunks
0 output ECNE chunks
0 output AUTH chunks
0 ip_output error counter
Packet drop statistics:
0 from middle box
0 from end host
0 with data
0 non-data, non-endhost
0 non-endhost, bandwidth rep only
0 not enough for chunk header
0 not enough data to confirm
0 where process_chunk_drop said break
0 failed to find TSN
0 attempt reverse TSN lookup
0 e-host confirms zero-rwnd
0 midbox confirms no space
0 data did not match TSN
0 TSN's marked for Fast Retran
Timeouts:
0 iterator timers fired
0 T3 data time outs
0 window probe (T3) timers fired
0 INIT timers fired
0 sack timers fired
0 shutdown timers fired
0 heartbeat timers fired
0 a cookie timeout fired
0 an endpoint changed its cookiesecret
0 PMTU timers fired
0 shutdown ack timers fired
0 shutdown guard timers fired
0 stream reset timers fired
0 early FR timers fired
0 an asconf timer fired
0 auto close timer fired
0 asoc free timers expired
0 inp free timers expired
0 packet shorter than header
0 checksum error
0 no endpoint for port
0 bad v-tag
0 bad SID
0 no memory
0 number of multiple FR in a RTT window
0 RFC813 allowed sending
0 RFC813 does not allow sending
0 times max burst prohibited sending
0 look ahead tells us no memory in interface
0 numbers of window probes sent
0 times an output error to clamp down on next user send
0 times sctp_senderrors were caused from a user
0 number of in data drops due to chunk limit reached
0 number of in data drops due to rwnd limit reached
0 times a ECN reduced the cwnd
0 used express lookup via vtag
0 collision in express lookup
0 times the sender ran dry of user data on primary
0 same for above
0 sacks the slow way
0 window update only sacks sent
0 sends with sinfo_flags !=0
0 unordered sends
0 sends with EOF flag set
0 sends with ABORT flag set
0 times protocol drain called
0 times we did a protocol drain
0 times recv was called with peek
0 cached chunks used
0 cached stream oq's used
0 unread messages abandonded by close
0 send burst avoidance, already max burst inflight to net
0 send cwnd full avoidance, already max burst inflight to net
0 number of map array over-runs via fwd-tsn's
ip:
21201 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
21072 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
24519 packets sent from this host
2224 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
4 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
icmp:
65 calls to icmp_error
0 errors not generated in response to an icmp message
Output histogram:
echo reply: 5073
destination unreachable: 65
0 messages with bad code fields
0 messages less than the minimum length
0 messages with bad checksum
0 messages with bad length
0 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echo reply: 117
destination unreachable: 51
echo: 5073
time exceeded: 4136
5073 message responses generated
0 invalid return addresses
0 no return routes
ICMP address mask responses are disabled
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with wrong TTL
0 messages received with bad checksum
0 V1/V2 membership queries received
0 V3 membership queries received
0 membership queries received with invalid field(s)
0 general queries received
0 group queries received
0 group-source queries received
0 group-source queries dropped
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 V3 reports received without Router Alert
0 membership reports sent
ipsec:
0 inbound packets violated process security policy
0 inbound packets failed due to insufficient memory
0 invalid inbound packets
0 outbound packets violated process security policy
0 outbound packets with no SA available
0 outbound packets failed due to insufficient memory
0 outbound packets with no route available
0 invalid outbound packets
0 outbound packets with bundled SAs
0 spd cache hits
0 spd cache misses
0 clusters copied during clone
0 mbufs inserted during makespace
ah:
0 packets shorter than header shows
0 packets dropped; protocol family not supported
0 packets dropped; no TDB
0 packets dropped; bad KCR
0 packets dropped; queue full
0 packets dropped; no transform
0 replay counter wraps
0 packets dropped; bad authentication detected
0 packets dropped; bad authentication length
0 possible replay packets detected
0 packets in
0 packets out
0 packets dropped; invalid TDB
0 bytes in
0 bytes out
0 packets dropped; larger than IP_MAXPACKET
0 packets blocked due to policy
0 crypto processing failures
0 tunnel sanity check failures
esp:
0 packets shorter than header shows
0 packets dropped; protocol family not supported
0 packets dropped; no TDB
0 packets dropped; bad KCR
0 packets dropped; queue full
0 packets dropped; no transform
0 packets dropped; bad ilen
0 replay counter wraps
0 packets dropped; bad encryption detected
0 packets dropped; bad authentication detected
0 possible replay packets detected
0 packets in
0 packets out
0 packets dropped; invalid TDB
0 bytes in
0 bytes out
0 packets dropped; larger than IP_MAXPACKET
0 packets blocked due to policy
0 crypto processing failures
0 tunnel sanity check failures
ipcomp:
0 packets shorter than header shows
0 packets dropped; protocol family not supported
0 packets dropped; no TDB
0 packets dropped; bad KCR
0 packets dropped; queue full
0 packets dropped; no transform
0 replay counter wraps
0 packets in
0 packets out
0 packets dropped; invalid TDB
0 bytes in
0 bytes out
0 packets dropped; larger than IP_MAXPACKET
0 packets blocked due to policy
0 crypto processing failures
0 packets sent uncompressed; size < compr. algo. threshold
0 packets sent uncompressed; compression was useless
arp:
35 ARP requests sent
2 ARP replies sent
2 ARP requests received
11 ARP replies received
13 ARP packets received
1 total packet dropped due to no ARP entry
4 ARP entrys timed out
0 Duplicate IPs seen
ip6:
21 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
19 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
59 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input histogram:
TCP: 2
UDP: 10
ICMP6: 9
Mbuf statistics:
11 one mbuf
10 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not contiguous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
3 failures of source address selection
source addresses on an outgoing I/F
2 link-locals
13 globals
source addresses on a non-outgoing I/F
3 addresses scope=0xf
source addresses of same scope
2 link-locals
3 globals
source addresses of a different scope
10 globals
Source addresses selection rule applied:
15 first candidate
3 same address
3 appropriate scope
icmp6:
0 calls to icmp6_error
0 errors not generated in response to an icmp6 message
0 errors not generated because of rate limitation
Output histogram:
echo: 3
echo reply: 3
neighbor solicitation: 29
MLDv2 listener report: 6
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
echo: 3
echo reply: 3
neighbor advertisement: 3
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
3 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
ipsec6:
0 inbound packets violated process security policy
0 inbound packets failed due to insufficient memory
0 invalid inbound packets
0 outbound packets violated process security policy
0 outbound packets with no SA available
0 outbound packets failed due to insufficient memory
0 outbound packets with no route available
0 invalid outbound packets
0 outbound packets with bundled SAs
0 spd cache hits
0 spd cache misses
0 clusters copied during clone
0 mbufs inserted during makespace
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output