Packet Filter max-src-conn and synproxy state

rambetter

rambetter

I'm following this guide to set up OpenBSD Packet Filter rules on my FreeBSD 8.0 system:
http://www.openbsd.org/faq/pf/filter.html

My goal is to limit the number of concurrent TCP connections from any given IP address to my webserver to 10 connections. I started with the following rules in my /etc/pf.conf, which worked fairly well (but I will describe the problem later):

Code: 
ext_if = "em0"
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
  keep state (source-track rule, max-src-conn 10)

When I load these rules and try to connect 100 TCP sockets to my webserver, the webserver (httpd) still spawns 100 child processes to handle the 100 incoming TCP connections. Also during the few seconds that the 100 connections are made, I ran ``pfctl -s state'' to check out the PF state table. This is what I got:

Code: 
all tcp 64.156.193.115:80 <- 64.156.192.169:62709       ESTABLISHED:ESTABLISHED
...
<10 lines of ESTABLISHED>
...
all tcp 64.156.193.115:80 <- 64.156.192.169:56010       ESTABLISHED:ESTABLISHED
all tcp 64.156.193.115:80 <- 64.156.192.169:55621       CLOSED:CLOSED
...
<very many lines of CLOSED>
all tcp 64.156.193.115:80 <- 64.156.192.169:55621       CLOSED:CLOSED

Apparently (and this is my guess), PF closed the connections once the 3-way handshake was made. However, since the 3-way handshake was made, Apache did spawn child processes to handle these requests. Now that is the very thing I'm trying to avoid - I don't want Apache to spawn more than 10 child processes for any given IP address that it's handling.


OK, so my next step is to try to use PF's TCP SYN Proxy. Sounds reasonable, right? This way, PF will do the handshake itself without passing the SYN packet to the Apache webserver (delay the handshake with Apache until later). In order to just try out the SYN proxy without trying to limit the connections from one IP address to 10, I wrote a simple pf.conf ruleset like so:

Code: 
ext_if = "em0"
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
  synproxy state

Now, when I load this ruleset and attempt a single TCP connection to the webserver, I get this in the output of my ``pfctl -s state'' command:

Code: 
all tcp 64.156.193.115:80 <- 99.50.206.241:37880       PROXY:DST

The person trying to connect via TCP to the webserver just waits forever.

I believe that the overall correct approach to what I'm trying to do is to get synproxy to work. And then use the max-src-conn syntax to only allow 10 concurrent connections per IP address. Am I correct?

How do I get synproxy to work? Should I ask about this on the OpenBSD forums since Packet Filter is from OpenBSD? Where else can I ask about this?
 
OK I think I just answered my own question. I added ``if-bound'' to my state clause in pf.conf. My entire ruleset is now this:

Code: 
ext_if = "em0"
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
  synproxy state (source-track rule, max-src-conn 10, if-bound)

.. and it works beautifully.
 
I did find something not quite right with my new setup. Here is what I did:

1. Load the pf.conf ruleset above (on server running Apache).

2. Clear the cache on my browser, close it, and open it again. I want the browser to make TCP connections for all the resources on the website.

3. Hit the home page of the website (which is behind the PF firewall, rules stated above).

4. Look at the output of ``pfctl -s state'' on the server machine, I see:
Code: 
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59970       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59971       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59972       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59973       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59974       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59975       ESTABLISHED:ESTABLISHED
Six TCP connections for getting the various resources from the homepage, fine. Keep-alive being used, fine.

5. I wait a few seconds, until I see this with ``pfctl -s state'':
Code: 
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59963       FIN_WAIT_2:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59964       FIN_WAIT_2:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59965       FIN_WAIT_2:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59967       FIN_WAIT_2:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59968       FIN_WAIT_2:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59969       FIN_WAIT_2:ESTABLISHED
Apparently the connections from the browser hit the Keep-alive timeout, or something like that happened. Why does it say ESTABLISHED to the left of the colon?

6. At this point, I hit a link on the website that takes me to a different page on the same site. Now I see this with ``pfctl -s state'':
Code: 
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59963       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59964       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59965       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59967       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59968       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59969       FIN_WAIT_2:FIN_WAIT_2
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59970       ESTABLISHED:ESTABLISHED
em0 tcp 64.156.193.115:80 <- 99.50.206.241:59971       ESTABLISHED:ESTABLISHED

7. If I keep waiting for the ESTABLISHED:ESTABLISHED to turn into FIN_WAIT_2:ESTABLISHED and then hit a new page that is not yet in the browser cache, new ESTABLISHED:ESTABLISHED entries will pop up in the state table. And they keep growing in this way, until I hit the max connections allowed by the firewall per host. The FIN_WAIT_2:FIN_WAIT_2 entries do go away eventually, but it takes a minute or so.


Would it be correct to configure something in PF to have the FIN_WAIT_2:FIN_WAIT_2 states go away sooner, or maybe have the FIN_WAIT_2:ESTABLISHED states go away sooner? How about not counting the FIN_WAIT_2:FIN_WAIT_2 states as open connections (for purposes of computing the number of connections allowed per host in my firewall)? How would I do that? Any other suggestions? What do these states mean?
 
General (tcp open / tcp close):

http://www.freesoft.org/CIE/Course/Section4/10.htm
http://www.freesoft.org/CIE/Course/Section4/11.htm
http://www.ssfnet.org/Exchange/tcp/tcpTutorialNotes.html (state diagram)

Specific:

max-src-conn 10 is pretty low for a webserver. Some webbrowsers can be instructed to open 16 http connections at once to streamline page loading, and visitors who like to click on links as soon as they appear may take up some more. I use '40' myself.

The FIN_WAIT stuff you see is pretty normal for a webbrowser <-> webserver interaction. HTTP connections are extremely shortlived ('give me pic.jpg, thanks, bye' -- 'give me image.png, thanks, bye' -- 'give me page.html, thanks, bye' (simultaneously)) and most of a 'web visit session' consists of setting up and tearing down these very short tcp connections, which is why you see a (perceived) state inconsistency.

So long as you allow some breathing room for these sessions to be 'established, but already dying', you should be fine. But '10' is not enough breathing room, I think.
 
Thanks DutchDaemon.

I did some TCP-related experiments by writing a couple of simple programs (one is a client and the other is a server). I'm beginning to understand this protocol more and what FIN_WAIT_2 means.

SIMPLE PROGRAM
SCENARIO #1:
Server listens for incoming sockets.
Client connects.
Client sends short message.
Server reads message.
Server sends short response message.
Server closes socket.
Client reads respnse message.
Client closes socket.

The above scenario will result in a FIN_WAIT_2:FIN_WAIT_2 state.


SCENARIO #2:
Server listens for incoming sockets.
Client connects.
Client sends short message.
Server reads message.
Server sends short response message.
Server does not close socket, goes to sleep.
Client reads response message.
Client closes socket.

The above scenario will result in an ESTABLISHED:FIN_WAIT_2 state.


SCENARIO #3:
Server listens for incoming sockets.
Client connects.
Client sends short message.
Server reads message.
Server sends short response message.
Server closes socket.
Client reads response message.
Client does not close socket, goes to sleep.

The above scenario will result in a FIN_WAIT_2:ESTABLISHED state.


So apparently the state means SERVER_STATE:CLIENT_STATE

In all 3 scenarios, the state will remain in PF's state table for a minute or so. This state will be counted towards the max-src-conn for the IP address. I didn't realize until now that the FIN_WAIT_2:FIN_WAIT_2 is considered to be an open connection with respect to calculating max-src-conn, and that it lingers for some time.

Is there a way to have max-src-conn only count connections that are considered to be ESTABLISHED on the SERVER_STATE?

My end goal is to limit the number of Apache TCP concurrent connections from one client in order to prevent an easy Slowloris attack from one of a small number of clients.

Actually the only person who has attacked my webserver with the Slowloris attack is myself. I'm trying to learn how to write robust TCP services; that is the underlying point of this exercise.
 
Is there any way to clear a state when a FIN packet is sent from the server? For example:

Code: 
pass out on $ext_if proto tcp from any port = http to any flags F/F \
  <remove state for this connection>

I could not find anything like this. I really want to limit the number of active TCP connections only.
 
pfctl -sa will show current timeout values (90 seconds for 'closed' is not that long). You could try setting pf's optimization to aggressive. The only other way to kill states is scripting something calling pfctl -k.
 
Hi

I am having a problem with max-src-conn and states as well, currently I have the limit for max-src-conn on port 80 set to 80 which I did think was conservative.

I then loaded a web page hosted on my server which resulted in timeouts on various images.

pfctl -ss yeilded this for my ip (my browser is set to 8 connections).

ips edited out.

Code: 
all tcp server:80 <- me:56206       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56209       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56210       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56221       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56222       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56223       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56224       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56225       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56226       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56227       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56228       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56229       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56230       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56231       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56232       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56233       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56234       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56235       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56236       TIME_WAIT:TIME_WAIT
all tcp server:80 <- me:56237       TIME_WAIT:TIME_WAIT

so (a) there is less than 80 there and (b) does max-src-conn count time_wait as connected?

I see the problem only occurs without keepalive so when browser (in this case IE) has http 1.1 disabled. However I have to assume some people will be using http 1.0 or have keepalive off in whatever other browser they use so if possible would appreciate a workaround, for http 1.0 to work smoothly I had to set a 100 limit and this is on a site with only 9 images.
 
You must log in or register to reply here.
Back
Top