Nginx causes Limiting open port RST response from 69582 to 200 packets/sec

Hi

I have the server running for a year. This just happened tonight:
- The server's networking is interrupted
- Log to see /var/log/messages, I got:
Code:
Nov 19 00:54:00 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:01 m153 kernel: Limiting open port RST response from 72183 to 200 packets/sec
Nov 19 00:54:01 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:02 m153 kernel: Limiting open port RST response from 68956 to 200 packets/sec
Nov 19 00:54:02 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:03 m153 kernel: Limiting open port RST response from 68586 to 200 packets/sec
Nov 19 00:54:03 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:04 m153 kernel: Limiting open port RST response from 69121 to 200 packets/sec
Nov 19 00:54:04 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:05 m153 kernel: Limiting open port RST response from 68789 to 200 packets/sec
Nov 19 00:54:05 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:06 m153 kernel: Limiting open port RST response from 70029 to 200 packets/sec
Nov 19 00:54:06 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:07 m153 kernel: Limiting open port RST response from 69507 to 200 packets/sec
Nov 19 00:54:07 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:08 m153 kernel: Limiting open port RST response from 69730 to 200 packets/sec
Nov 19 00:54:08 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:09 m153 kernel: Limiting open port RST response from 69542 to 200 packets/sec
Nov 19 00:54:09 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:10 m153 kernel: Limiting open port RST response from 69227 to 200 packets/sec
Nov 19 00:54:10 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule

- I have set the /etc/sysctl.conf as below and restart the system:
Code:
# General Security and DoS mitigation.
net.inet.ip.check_interface=1         # verify packet arrives on correct interface (default 0)
net.inet.ip.portrange.randomized=1    # randomize outgoing upper ports (default 1)
net.inet.ip.process_options=0         # IP options in the incoming packets will be ignored (default 1)
net.inet.ip.random_id=1               # assign a random IP_ID to each packet leaving the system (default 0)
net.inet.ip.redirect=0                # do not send IP redirects (default 1)
net.inet.ip.accept_sourceroute=0      # drop source routed packets since they can not be trusted (default 0)
net.inet.ip.sourceroute=0             # if source routed packets are accepted the route data is ignored (default 0)
net.inet.icmp.bmcastecho=0            # do not respond to ICMP packets sent to IP broadcast addresses (default 0)
net.inet.icmp.maskfake=0              # do not fake reply to ICMP Address Mask Request packets (default 0)
net.inet.icmp.maskrepl=0              # replies are not sent for ICMP address mask requests (default 0)
net.inet.icmp.log_redirect=0          # do not log redirected ICMP packet attempts (default 0)
net.inet.icmp.drop_redirect=1         # no redirected ICMP packets (default 0)
net.inet.icmp.icmplim_output=1        # show "Limiting open port RST response" messages (default 1)
net.inet.tcp.always_keepalive=0       # tcp keep alive detection for dead peers, can be spoofed (default 1)
net.inet.tcp.drop_synfin=1            # SYN/FIN packets get dropped on initial connection (default 0)
#net.inet.tcp.fast_finwait2_recycle=1  # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0)
net.inet.tcp.icmp_may_rst=0           # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.msl=15000                # 15s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0     # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
net.inet.tcp.rfc3042=0                # disable limited transmit mechanism which can slow burst transmissions (default 1)
net.inet.tcp.sack.enable=1            # TCP Selective Acknowledgments are needed for high throughput (default 1)
net.inet.udp.blackhole=1              # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2              # drop tcp packets destined for closed ports (default 0)

- When I turn off the Nginx process, the /var/log/messages does not show any above messages. Turning on Nginx they appear again.

- When turn off Nginx, the system still report high usage. With top command:
Code:
last pid:  1979;  load averages:  1.18,  1.17,  1.01                                                                                      up 0+00:37:55  01:13:28
30 processes:  1 running, 29 sleeping
CPU:  0.0% user,  0.0% nice,  8.3% system,  0.0% interrupt, 91.6% idle
Mem: 532M Active, 4852M Inact, 1030M Wired, 824M Buf, 1498M Free
Swap: 15G Total, 15G Free

- The system is:
Code:
#uname -a
FreeBSD m153.admansend.com 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

I've been searching solution for hours without success. Please advise on solving this problem.
 
The server is being attacked by SYN flood.
Code:
01:14:35.528497 IP 17.123.184.146.3530 > m.domain.com.http: Flags |S|, seq 2461563615, win 41091, length 0
01:14:35.528498 IP 5.85.196.234.16330 > m.domain.com.http: Flags |S|, seq 3938735280, win 10897, length 0
01:14:35.528500 IP 63.221.104.45.2728 > m.domain.com.http: Flags |S|, seq 761846940, win 14353, length 0
01:14:35.528501 IP 141.128.26.37.42407 > m.domain.com.http: Flags |S|, seq 622493761, win 51905, length 0
01:14:35.528502 IP 55.143.54.2.37229 > m.domain.com.http: Flags |S|, seq 37129935, win 37795, length 0
01:14:35.528503 IP 77.77.210.255.19744 > m.domain.com.http: Flags |S|, seq 4291972411, win 19701, length 0
01:14:35.528505 IP host86-152-40-52.range86-152.btcentralplus.com.52350 > m.domain.com.http: Flags |S|, seq 875075595, win 168, length 0
01:14:35.528507 IP 119.100.69.38.35516 > m.domain.com.http: Flags |S|, seq 642081826, win 45313, length 0
01:14:35.528508 IP 50.214.250.32.63276 > m.domain.com.http: Flags |S|, seq 553309620, win 6184, length 0
01:14:35.528509 IP c-67-172-176-246.hsd1.ca.comcast.net.41716 > m.domain.com.http: Flags |S|, seq 4138773525, win 39335, length 0
01:14:35.528511 IP 168.32.202.56.22898 > m.domain.com.http: Flags |S|, seq 952770630, win 37436, length 0
01:14:35.528512 IP 174-29-59-224.hlrn.qwest.net.65001 > m.domain.com.http: Flags |S|, seq 3761970390, win 56870, length 0
01:14:35.528513 IP 48.19.239.186.52767 > m.domain.com.http: Flags |S|, seq 3136230090, win 35088, length 0
01:14:35.528514 IP 198.183.184.84.3199 > m.domain.com.http: Flags |S|, seq 1421391720, win 24888, length 0
01:14:35.528516 IP 32.76.76.154.58988 > m.domain.com.http: Flags |S|, seq 2588691285, win 32954, length 0
01:14:35.528517 IP 159.233.239.219.50575 > m.domain.com.http: Flags |S|, seq 3689933010, win 41345, length 0
01:14:35.528519 IP 191.156.74.203.26634 > m.domain.com.http: Flags |S|, seq 3410664480, win 13143, length 0
01:14:35.528520 IP 0.21.171.204.57771 > m.domain.com.http: Flags |S|, seq 3433763955, win 44632, length 0
01:14:35.528521 IP 2-249-146-45-no267.digitaltv.telia.com.9877 > m.domain.com.http: Flags |S|, seq 764606595, win 21544, length 0
01:14:35.528523 IP 29.218.119.223.47509 > m.domain.com.http: Flags |S|, seq 3749173650, win 39183, length 0
01:14:35.528524 IP 173-18-16-123.client.mchsi.com.36285 > m.domain.com.http: Flags |S|, seq 2064650775, win 2255, length 0
01:14:35.528525 IP 252.92.160.99.49308 > m.domain.com.http: Flags |S|, seq 1671453930, win 9278, length 0
01:14:35.528527 IP 169.116.172.157.4694 > m.domain.com.http: Flags |S|, seq 2645324760, win 45059, length 0
01:14:35.528528 IP 13.157.255.215.29965 > m.domain.com.http: Flags |S|, seq 3623853090, win 19727, length 0
01:14:35.528530 IP 201-187-167-15.bam.movistar.cl.52080 > m.domain.com.http: Flags |S|, seq 262650570, win 56087, length 0
01:14:35.528531 IP 36.220.111.201.42388 > m.domain.com.http: Flags |S|, seq 3379551256, win 28422, length 0
01:14:35.528532 IP 108.208.250.221.44647 > m.domain.com.http: Flags |S|, seq 3724201860, win 35940, length 0
01:14:35.528534 IP 95.162.191.87.64030 > m.domain.com.http: Flags |S|, seq 1472176666, win 20959, length 0
01:14:35.528535 IP 117.30.20.95.32137 > m.domain.com.http: Flags |S|, seq 1595153986, win 56477, length 0
01:14:35.528536 IP 32.113.118.4.30102 > m.domain.com.http: Flags |S|, seq 74870865, win 31244, length 0
01:14:35.528537 IP 39.118.167.12.33486 > m.domain.com.http: Flags |S|, seq 212301270, win 36725, length 0
01:14:35.528539 IP 178.188.118.150.21289 > m.domain.com.http: Flags |S|, seq 2524363876, win 59808, length 0
01:14:35.528541 IP 197.6.3.136.36552 > m.domain.com.http: Flags |S|, seq 2281899690, win 5837, length 0
01:14:35.528542 IP 251.148.205.115.2249 > m.domain.com.http: Flags |S|, seq 1942852665, win 31895, length 0
 
Back
Top