• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Nginx causes Limiting open port RST response from 69582 to 200 packets/sec

hieutmd

New Member


Messages: 8

#1
Hi

I have the server running for a year. This just happened tonight:
- The server's networking is interrupted
- Log to see /var/log/messages, I got:
Code:
Nov 19 00:54:00 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:01 m153 kernel: Limiting open port RST response from 72183 to 200 packets/sec
Nov 19 00:54:01 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:02 m153 kernel: Limiting open port RST response from 68956 to 200 packets/sec
Nov 19 00:54:02 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:03 m153 kernel: Limiting open port RST response from 68586 to 200 packets/sec
Nov 19 00:54:03 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:04 m153 kernel: Limiting open port RST response from 69121 to 200 packets/sec
Nov 19 00:54:04 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:05 m153 kernel: Limiting open port RST response from 68789 to 200 packets/sec
Nov 19 00:54:05 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:06 m153 kernel: Limiting open port RST response from 70029 to 200 packets/sec
Nov 19 00:54:06 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:07 m153 kernel: Limiting open port RST response from 69507 to 200 packets/sec
Nov 19 00:54:07 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:08 m153 kernel: Limiting open port RST response from 69730 to 200 packets/sec
Nov 19 00:54:08 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:09 m153 kernel: Limiting open port RST response from 69542 to 200 packets/sec
Nov 19 00:54:09 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
Nov 19 00:54:10 m153 kernel: Limiting open port RST response from 69227 to 200 packets/sec
Nov 19 00:54:10 m153 kernel: ipfw: add_dyn_rule: Cannot allocate rule
- I have set the /etc/sysctl.conf as below and restart the system:
Code:
# General Security and DoS mitigation.
net.inet.ip.check_interface=1         # verify packet arrives on correct interface (default 0)
net.inet.ip.portrange.randomized=1    # randomize outgoing upper ports (default 1)
net.inet.ip.process_options=0         # IP options in the incoming packets will be ignored (default 1)
net.inet.ip.random_id=1               # assign a random IP_ID to each packet leaving the system (default 0)
net.inet.ip.redirect=0                # do not send IP redirects (default 1)
net.inet.ip.accept_sourceroute=0      # drop source routed packets since they can not be trusted (default 0)
net.inet.ip.sourceroute=0             # if source routed packets are accepted the route data is ignored (default 0)
net.inet.icmp.bmcastecho=0            # do not respond to ICMP packets sent to IP broadcast addresses (default 0)
net.inet.icmp.maskfake=0              # do not fake reply to ICMP Address Mask Request packets (default 0)
net.inet.icmp.maskrepl=0              # replies are not sent for ICMP address mask requests (default 0)
net.inet.icmp.log_redirect=0          # do not log redirected ICMP packet attempts (default 0)
net.inet.icmp.drop_redirect=1         # no redirected ICMP packets (default 0)
net.inet.icmp.icmplim_output=1        # show "Limiting open port RST response" messages (default 1)
net.inet.tcp.always_keepalive=0       # tcp keep alive detection for dead peers, can be spoofed (default 1)
net.inet.tcp.drop_synfin=1            # SYN/FIN packets get dropped on initial connection (default 0)
#net.inet.tcp.fast_finwait2_recycle=1  # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0)
net.inet.tcp.icmp_may_rst=0           # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.msl=15000                # 15s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0     # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
net.inet.tcp.rfc3042=0                # disable limited transmit mechanism which can slow burst transmissions (default 1)
net.inet.tcp.sack.enable=1            # TCP Selective Acknowledgments are needed for high throughput (default 1)
net.inet.udp.blackhole=1              # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2              # drop tcp packets destined for closed ports (default 0)
- When I turn off the Nginx process, the /var/log/messages does not show any above messages. Turning on Nginx they appear again.

- When turn off Nginx, the system still report high usage. With top command:
Code:
last pid:  1979;  load averages:  1.18,  1.17,  1.01                                                                                      up 0+00:37:55  01:13:28
30 processes:  1 running, 29 sleeping
CPU:  0.0% user,  0.0% nice,  8.3% system,  0.0% interrupt, 91.6% idle
Mem: 532M Active, 4852M Inact, 1030M Wired, 824M Buf, 1498M Free
Swap: 15G Total, 15G Free
- The system is:
Code:
#uname -a
FreeBSD m153.admansend.com 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
I've been searching solution for hours without success. Please advise on solving this problem.
 

hieutmd

New Member


Messages: 8

#2
The server is being attacked by SYN flood.
Code:
01:14:35.528497 IP 17.123.184.146.3530 > m.domain.com.http: Flags |S|, seq 2461563615, win 41091, length 0
01:14:35.528498 IP 5.85.196.234.16330 > m.domain.com.http: Flags |S|, seq 3938735280, win 10897, length 0
01:14:35.528500 IP 63.221.104.45.2728 > m.domain.com.http: Flags |S|, seq 761846940, win 14353, length 0
01:14:35.528501 IP 141.128.26.37.42407 > m.domain.com.http: Flags |S|, seq 622493761, win 51905, length 0
01:14:35.528502 IP 55.143.54.2.37229 > m.domain.com.http: Flags |S|, seq 37129935, win 37795, length 0
01:14:35.528503 IP 77.77.210.255.19744 > m.domain.com.http: Flags |S|, seq 4291972411, win 19701, length 0
01:14:35.528505 IP host86-152-40-52.range86-152.btcentralplus.com.52350 > m.domain.com.http: Flags |S|, seq 875075595, win 168, length 0
01:14:35.528507 IP 119.100.69.38.35516 > m.domain.com.http: Flags |S|, seq 642081826, win 45313, length 0
01:14:35.528508 IP 50.214.250.32.63276 > m.domain.com.http: Flags |S|, seq 553309620, win 6184, length 0
01:14:35.528509 IP c-67-172-176-246.hsd1.ca.comcast.net.41716 > m.domain.com.http: Flags |S|, seq 4138773525, win 39335, length 0
01:14:35.528511 IP 168.32.202.56.22898 > m.domain.com.http: Flags |S|, seq 952770630, win 37436, length 0
01:14:35.528512 IP 174-29-59-224.hlrn.qwest.net.65001 > m.domain.com.http: Flags |S|, seq 3761970390, win 56870, length 0
01:14:35.528513 IP 48.19.239.186.52767 > m.domain.com.http: Flags |S|, seq 3136230090, win 35088, length 0
01:14:35.528514 IP 198.183.184.84.3199 > m.domain.com.http: Flags |S|, seq 1421391720, win 24888, length 0
01:14:35.528516 IP 32.76.76.154.58988 > m.domain.com.http: Flags |S|, seq 2588691285, win 32954, length 0
01:14:35.528517 IP 159.233.239.219.50575 > m.domain.com.http: Flags |S|, seq 3689933010, win 41345, length 0
01:14:35.528519 IP 191.156.74.203.26634 > m.domain.com.http: Flags |S|, seq 3410664480, win 13143, length 0
01:14:35.528520 IP 0.21.171.204.57771 > m.domain.com.http: Flags |S|, seq 3433763955, win 44632, length 0
01:14:35.528521 IP 2-249-146-45-no267.digitaltv.telia.com.9877 > m.domain.com.http: Flags |S|, seq 764606595, win 21544, length 0
01:14:35.528523 IP 29.218.119.223.47509 > m.domain.com.http: Flags |S|, seq 3749173650, win 39183, length 0
01:14:35.528524 IP 173-18-16-123.client.mchsi.com.36285 > m.domain.com.http: Flags |S|, seq 2064650775, win 2255, length 0
01:14:35.528525 IP 252.92.160.99.49308 > m.domain.com.http: Flags |S|, seq 1671453930, win 9278, length 0
01:14:35.528527 IP 169.116.172.157.4694 > m.domain.com.http: Flags |S|, seq 2645324760, win 45059, length 0
01:14:35.528528 IP 13.157.255.215.29965 > m.domain.com.http: Flags |S|, seq 3623853090, win 19727, length 0
01:14:35.528530 IP 201-187-167-15.bam.movistar.cl.52080 > m.domain.com.http: Flags |S|, seq 262650570, win 56087, length 0
01:14:35.528531 IP 36.220.111.201.42388 > m.domain.com.http: Flags |S|, seq 3379551256, win 28422, length 0
01:14:35.528532 IP 108.208.250.221.44647 > m.domain.com.http: Flags |S|, seq 3724201860, win 35940, length 0
01:14:35.528534 IP 95.162.191.87.64030 > m.domain.com.http: Flags |S|, seq 1472176666, win 20959, length 0
01:14:35.528535 IP 117.30.20.95.32137 > m.domain.com.http: Flags |S|, seq 1595153986, win 56477, length 0
01:14:35.528536 IP 32.113.118.4.30102 > m.domain.com.http: Flags |S|, seq 74870865, win 31244, length 0
01:14:35.528537 IP 39.118.167.12.33486 > m.domain.com.http: Flags |S|, seq 212301270, win 36725, length 0
01:14:35.528539 IP 178.188.118.150.21289 > m.domain.com.http: Flags |S|, seq 2524363876, win 59808, length 0
01:14:35.528541 IP 197.6.3.136.36552 > m.domain.com.http: Flags |S|, seq 2281899690, win 5837, length 0
01:14:35.528542 IP 251.148.205.115.2249 > m.domain.com.http: Flags |S|, seq 1942852665, win 31895, length 0