I have a problem with delay 3 seconds when using proxying to backend. For proxying uses nginx or haproxy.
No cases 1 sec or 2 sec. Only 3 sec. On backend problem requests work less 0.1 sec.
If sending directly to backend, 3 sec delays is not.
The proportion of such requests is ~0.1%
OS - 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
ipwf not uses
Scheme
BSD(nginx)->Linux(java app)
=config sysctl=
=config loader=
I found same problem here
http://mailing.freebsd.net.narkive.com/Irqiac25/random-tcp-3-second-delay
and here
https://groups.google.com/forum/#!topic/liftweb/ddtUdWJ0kGI
but theese solutions not work for me.
vmstat -z
No cases 1 sec or 2 sec. Only 3 sec. On backend problem requests work less 0.1 sec.
If sending directly to backend, 3 sec delays is not.
The proportion of such requests is ~0.1%
OS - 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
ipwf not uses
Scheme
BSD(nginx)->Linux(java app)
=config sysctl=
Code:
net.inet.tcp.syncookies=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.msl=1000
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.ip.portrange.first=1024
net.inet.ip.portrange.last=65535
security.jail.set_hostname_allowed = 1
security.jail.allow_raw_sockets = 0
security.jail.chflags_allowed = 0
kern.ipc.somaxconn=2048
kern.ipc.maxsockets=204800
kern.maxfiles=204800
kern.maxvnodes=200000
kern.maxfilesperproc=200000
kern.maxprocperuid=262144
net.inet.tcp.finwait2_timeout=20000
net.inet.tcp.fast_finwait2_recycle=1
net.inet.carp.preempt= 1
net.inet.tcp.maxtcptw=30000
kern.ipc.soacceptqueue=4096=config loader=
Code:
vmxnet_load="YES"
vmxnet3_load="YES"
kern.racct.enable=1
carp_load="YES"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=128
net.inet.tcp.syncache.cachelimit=65536I found same problem here
http://mailing.freebsd.net.narkive.com/Irqiac25/random-tcp-3-second-delay
and here
https://groups.google.com/forum/#!topic/liftweb/ddtUdWJ0kGI
but theese solutions not work for me.
vmstat -z
Code:
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 384, 0, 107, 3, 107, 0, 0
UMA Zones: 1664, 0, 107, 1, 107, 0, 0
UMA Slabs: 80, 0, 2366, 34, 2681, 0, 0
UMA RCntSlabs: 88, 0, 2784, 6, 2784, 0, 0
UMA Hash: 256, 0, 2, 13, 8, 0, 0
4 Bucket: 32, 0, 37, 2338, 75332, 0, 0
6 Bucket: 48, 0, 22, 2551, 745894, 0, 0
8 Bucket: 64, 0, 52, 2490, 84919, 11, 0
12 Bucket: 96, 0, 112, 1528, 226802, 0, 0
16 Bucket: 128, 0, 148, 1495, 38035, 0, 0
32 Bucket: 256, 0, 532, 1493, 55856, 52, 0
64 Bucket: 512, 0, 241, 335, 209610, 280, 0
128 Bucket: 1024, 0, 577, 163, 125967, 464, 0
256 Bucket: 2048, 0, 312, 72, 164465, 0, 0
vmem btag: 56, 0, 1218, 10781, 20315, 85, 0
VM OBJECT: 256, 0, 53711, 2089,102349193, 0, 0
RADIX NODE: 144, 0, 15639, 1722,164401641, 0, 0
MAP: 240, 0, 3, 61, 3, 0, 0
KMAP ENTRY: 128, 0, 9, 394, 9, 0, 0
MAP ENTRY: 128, 0, 1615, 3655,236592556, 0, 0
VMSPACE: 448, 0, 52, 362, 3986647, 0, 0
fakepg: 104, 0, 0, 152, 6, 0, 0
mt_zone: 4112, 0, 371, 0, 371, 0, 0
16: 16, 0, 2839, 1679, 9739511, 0, 0
32: 32, 0, 6434, 11066,10976573, 0, 0
64: 64, 0, 6098, 3016,143452073, 0, 0
128: 128, 0, 8126, 2755, 4027041, 0, 0
256: 256, 0, 1105, 1895,30360316, 0, 0
512: 512, 0, 827, 1261, 544195, 0, 0
1024: 1024, 0, 128, 200, 3318869, 0, 0
2048: 2048, 0, 119, 143, 4886283, 0, 0
4096: 4096, 0, 449, 85, 4280908, 0, 0
8192: 8192, 0, 32, 10, 26306, 0, 0
16384: 16384, 0, 24, 50, 2780280, 0, 0
32768: 32768, 0, 17, 11, 1262, 0, 0
65536: 65536, 0, 2, 4, 279, 0, 0
SLEEPQUEUE: 80, 0, 271, 752, 271, 0, 0
64 pcpu: 8, 0, 1722, 838, 1722, 0, 0
Files: 80, 0, 4517, 1483,44228567, 0, 0
TURNSTILE: 136, 0, 271, 289, 271, 0, 0
rl_entry: 40, 0, 183, 1917, 183, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
rctl_rule_link: 32, 0, 0, 0, 0, 0, 0
rctl_rule: 80, 0, 0, 0, 0, 0, 0
racct: 176, 0, 78, 670, 4234701, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1264, 0, 69, 129, 3986663, 0, 0
THREAD: 1192, 0, 244, 26, 245, 0, 0
cpuset: 72, 0, 65, 485, 65, 0, 0
audit_record: 1248, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 1216065, 1013, 3541,52317832, 0, 0
mbuf: 256, 1216065, 7, 2279,17800623, 0, 0
mbuf_cluster: 2048, 190008, 4554, 12, 4554, 0, 0
mbuf_jumbo_page: 4096, 95004, 0, 501, 995659, 0, 0
mbuf_jumbo_9k: 9216, 28149, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 15834, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
ttyoutq: 256, 0, 95, 220, 198, 0, 0
g_bio: 248, 0, 0, 864, 821649, 0, 0
ttyinq: 160, 0, 180, 195, 375, 0, 0
DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0
nvme_request: 128, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
FPU_save_area: 832, 0, 0, 0, 0, 0, 0
VNODE: 472, 0, 84468, 796, 517379, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
BUF TRIE: 144, 0, 733, 19436, 131262, 0, 0
S VFS Cache: 108, 0, 56480, 35535, 7088135, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 28418, 562, 275555, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 156,141711573, 0, 0
DIRHASH: 1024, 0, 2292, 128, 4075, 0, 0
NCLNODE: 528, 0, 0, 0, 0, 0, 0
procdesc: 128, 0, 0, 0, 0, 0, 0
Mountpoints: 816, 0, 8, 22, 8, 0, 0
pipe: 744, 0, 5, 165, 3292707, 0, 0
ksiginfo: 112, 0, 189, 1701, 63024, 0, 0
itimer: 352, 0, 0, 0, 0, 0, 0
KNOTE: 128, 0, 1250, 1509, 8298209, 0, 0
socket: 696, 204800, 319, 191, 4573161, 0, 0
ipq: 56, 5964, 0, 0, 0, 0, 0
udp_inpcb: 392, 204800, 25, 385, 2152856, 0, 0
udpcb: 24, 204909, 25, 3649, 2152856, 0, 0
tcp_inpcb: 392, 204800, 3639, 9501, 302173, 0, 0
tcpcb: 1032, 204801, 212, 94, 302173, 0, 0
tcptw: 88, 30015, 3427, 10703, 135269, 0, 0
syncache: 160, 65550, 0, 500, 188039, 0, 0
hostcache: 136, 15370, 3, 142, 8, 0, 0
sackhole: 32, 0, 0, 375, 43, 0, 0
tcpreass: 40, 11900, 0, 300, 2, 0, 0
sctp_ep: 1424, 97520, 0, 0, 0, 0, 0
sctp_asoc: 2416, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80012, 0, 1328, 7, 0, 0
sctp_raddr: 728, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400026, 0, 0, 0, 0, 0
sctp_readq: 104, 400026, 0, 0, 0, 0, 0
sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400000, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0
udplite_inpcb: 392, 97520, 0, 0, 0, 0, 0
ripcb: 392, 204800, 0, 270, 987, 0, 0
unpcb: 240, 204800, 41, 695, 2117139, 0, 0
rtentry: 200, 0, 21, 299, 21, 0, 0
selfd: 56, 0, 219, 3118,13522347, 0, 0
SWAPMETA: 288, 380029, 0, 0, 0, 0, 0
FFS inode: 176, 0, 84429, 1041, 496802, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 84429, 1041, 496801, 0, 0