PF with ALTQ: when under synflood attack Nginx go offline

[raffo]

Hi,

I have rebuilt the kernel to enable PF and ALTQ by adding the following modules:

device pf
device pflog
device pfsync

options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build

My configuration of pf.conf is:

set optimization aggressive
set fingerprints "/etc/pf.os"
#set fingerprints 16384:64:0:64:M*,N,N,S,N,W0,N,N,T:OpenBSD:4.8:!df:OpenBSD 4.8 scrub no-df

set timeout interval 10
set timeout frag 30
set limit { frags 5000, states 2500 }
set block-policy drop
#set loginterface $int_if

#scrub
scrub in all

ext_if = "vr0"
set skip on lo0

pass quick on { lo, vr0, tap0, bridge0 }

antispoof for $ext_if inet
#match in all scrub (no-df)
block in quick from urpf-failed
antispoof log for $ext_if

#scrub in on $ext_if all fragment reassemble

# This will allow Slowloris attack from localhost, but that's OK.
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
  synproxy state (source-track rule, max-src-conn 10, if-bound)

#block return-rst in quick on $ext_if inet proto tcp from all

vpn_if="tun0"
AllowOUT="{80, 443}"
 TcpState="flags S/SAFR modulate state"

#nat on $ext_if from 10.8.0.0/24 to any -> ($ext_if) 
pass in log on vr0 inet proto tcp from vr0:network to !vr0 port $AllowOUT $TcpState #$ExtIfSTO

#nat on egress from vr0:network to any tag EGRESS -> (vr0:0) port 1024:65535

#ExtIfSTO="(max 9000, source-track rule, max-src-conn   2000, max-src-nodes 254)"

pass in quick proto udp from any to port 443 keep state label "openvpn"

# Pass stuff on the VPN interface
pass quick on $vpn_if keep state

block all
block drop in on ! vr0 inet from any to any
block out from any os "unknown"
block out from any os "NMAP"
block in log quick on $ext_if from any os "NMAP" to any label ExtNMAPScan
block in log quick on $ext_if inet proto tcp from any to any flags FUP/FUP
block out proto tcp from any os "NMAP	OS"
block out proto tcp from any os "NMAP	OS	1"
block out proto tcp from any os "NMAP	OS	2"
block out proto tcp from any os "NMAP   OS      3"
block out proto tcp from any os "NMAP   OS      4"
block out proto tcp from any os "NMAP	syn scan"
block out proto tcp from any os "NMAP	syn scan	1"
block out proto tcp from any os "NMAP   syn scan        2"
block out proto tcp from any os "NMAP   syn scan        3"
block out proto tcp from any os "NMAP   syn scan        4"

block out proto tcp from any os "*NMAP:OS:1:NMAP OS detection probe"
block out proto tcp from any os "*NMAP:OS:1:NMAP OS detection probe (1)"
block out proto tcp from any os "*NMAP:OS:1:NMAP OS detection probe (2)"
block out proto tcp from any os "*NMAP:OS:1:NMAP OS detection probe (3)"
block out proto tcp from any os "*NMAP:OS:1:NMAP OS detection probe (4)"

block out proto tcp from any os "*NAST:::NASTsyn scan"
block out proto tcp from any os "*NMAP:syn scan:1:NMAP syn scan (1)"
block out proto tcp from any os "*NMAP:syn scan:1:NMAP syn scan (2)"
block out proto tcp from any os "*NMAP:syn scan:1:NMAP syn scan (3)"
block out proto tcp from any os "*NMAP:syn scan:1:NMAP syn scan (4)"

block drop in log (all)  quick on $ext_if from <droplasso> to any
block drop out log (all) quick on $ext_if from any to <droplasso>

pass out on $ext_if inet proto udp from any to any port 33433 >< 33626 keep state

block in from no-route to any
block in from urpf-failed to any

block in quick on $ext_if proto tcp flags FUP/WEUAPRSF
block in quick on $ext_if proto tcp flags WEUAPRSF/WEUAPRSF
block in quick on $ext_if proto tcp flags SRAFU/WEUAPRSF
block in quick on $ext_if proto tcp flags /WEUAPRSF
block in quick on $ext_if proto tcp flags SR/SR
block in quick on $ext_if proto tcp flags SF/SF
block in quick on $ext_if proto tcp from any to any flags FUP/FUP

pass out on $ext_if proto {tcp, udp, icmp} from any to any modulate state

pass in on $ext_if proto tcp from any to any port ssh flags S/SA synproxy state
pass in on $ext_if proto tcp from any to any port www flags S/SA synproxy state
icmp_types="echoreq"
pass in inet proto icmp all icmp-type $icmp_types keep state
pass in quick proto icmp6 all

# ND solicitation out
pass out on $ext_if inet6 proto icmp6 all icmp6-type {neighbradv, neighbrsol}
 
# ND advertisement in
pass in on $ext_if inet6 proto icmp6 all icmp6-type {neighbradv, neighbrsol}
 
# Router advertisement out
pass out on $ext_if inet6 proto icmp6 all icmp6-type routeradv 
 
# Router solicitation in
pass in on $ext_if inet6 proto icmp6 all icmp6-type routersol
 
# Allow Ping pong in
pass in on $ext_if inet6 proto icmp6 all icmp6-type echoreq

#scrub in on $ext_if all fragment reassemble

pass in proto tcp from any to any \
		 port www keep state \
		 (max 100, source-track rule, max-src-nodes 120, \
		 max-src-states 3, tcp.established 500, tcp.closing 10)

#pass in on $ext_if proto tcp from any to any port http flags S/SA synproxy modulate state

#pass in on $ext_if proto tcp from any to $www port { 80, 443 } flags 
#S/SA keep state \ 
#(source-track rule, max-src-conn-rate 150/10, max-src-states 500, 
#max-src-nodes 4000000, overload <blockedips> flush global) 


table <bruteforce> persist
pass in proto tcp from any to any port ssh flags S/SA keep state \
    (source-track rule, max-src-conn-rate 2/10, overload <blockedips> flush global)
block drop in quick from <bruteforce> to any
block out quick from any to <bruteforce>

table <blockedips> persist file "/etc/pf.blocked.ip.conf"
block drop in log (all) quick on $ext_if from <blockedips> to any


# MySQL
pass in log on $ext_if proto tcp to port ssh synproxy state \
	(max-src-conn 100, max-src-conn-rate 15/5, overload <blockedips> flush)

table <ssh_abuse> persist
block in quick from <ssh_abuse>
pass in on $ext_if proto tcp to any port ssh flags S/SA keep state (max-src-conn 10, max-src-conn-rate 3/5, overload <blockedips> flush)

When I test my firewall with 16 servers that send 6000 pps, everything's fine with 145.000 pps in a 100mbps! But when one server of 1Gbps sends it 150.000 pps I just received 1000 pps and server are online, just Nginx go offline. The attack consists to open a lot of new connection syn_sent on port 80.

I tried a lot of config, I don't know how to limit the connections in nginx, when send connections and PF it block, I want nginx don't see anything.

Total connections was 9000, nginx can handle max 18.000 connection on this server.

So what is the problem? Is not a problem of nginx, is of PF that just don't block the connections.

Any suggestions?

Thanks

 

[quintessence]

Hello,

# sysctl net.inet.tcp.drop_synfin=1

18k connections is very low number, try to set lower values of all timeout directives, you can start with keepalive_timeout which by default on install from ports is 65 seconds (this is a lot if you serve small files).

However, you actually have take a look at OS level (sysctl variables for examples), not to Nginx directives.

http://people.freebsd.org/~jlemon/papers/syncache.pdf