IPFW IPFW not working

A

akshin

Hi!

IPFW config file
Code: 
# Block IP
ipfw table 1 add 115.0.0.0/8;
$cmd 160 deny ip from 'table(1)' to me
$cmd 161 deny ip from me to 'table(1)'

Code: 
root@mydomain:/var/logs/nginx# ipfw table 1 list
115.0.0.0/8 0

Why do I get lines like this in the NGINX error and access logs?

Code: 
2023/12/29 07:58:29 [error] 58617#100212: *433026219 limiting requests, excess: *.333 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:29 [error] 58620#100272: *433023057 limiting requests, excess: *.230 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:29 [error] 58619#100206: *433025527 limiting requests, excess: *.115 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58614#100255: *433021194 limiting requests, excess: *.517 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58617#100212: *433024814 limiting requests, excess: *.500 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58613#100259: *433026241 limiting requests, excess: *.484 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58617#100212: *433024619 limiting requests, excess: *.467 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58613#100259: *433027131 limiting requests, excess: *.450 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58613#100259: *433022189 limiting requests, excess: *.234 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:30 [error] 58613#100259: *433026996 limiting requests, excess: *.144 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:31 [error] 58614#100255: *433027291 limiting requests, excess: *.753 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:32 [error] 58613#100259: *433021182 limiting requests, excess: *.318 by zone "", client: 115.227.85.250, server: , request: "GET some url"
2023/12/29 07:58:32 [error] 58613#100259: *433021409 limiting requests, excess: *.203 by zone "", client: 115.227.85.250, server: , request: "GET some url"

NGINX acces log

Code: 
115.227.85.250 - - [29/Dec/2023:07:39:11 +0400] "GET /chn/bolme/official_chronicle HTTP/1.1" 200 12601 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
 
akshin said:
Hi!

IPFW config file
Code: 
# Block IP
ipfw table 1 add 115.0.0.0/8;
$cmd 160 deny ip from 'table(1)' to me
$cmd 161 deny ip from me to 'table(1)'
Click to expand...

Is that the entire ruleset? What is $cmd set to?

Make those lines do logging:
$cmd 160 deny log ip ...

And then tail /var/log/security to observe packets being blocked, with timestamps to be compared with nginx ones.

akshin said:
Code: 
root@mydomain:/var/logs/nginx# ipfw table 1 list
115.0.0.0/8 0
Click to expand...

Ok; Also result of:
# ipfw -t show

akshin said:
Why do I get lines like this in the NGINX error logs?

Code: 
2023/12/29 07:58:29 [error] 58617#100212: *433026219 limiting requests, excess: *.333 by zone "", client: 115.227.85.250, server: , request: "GET some url"
[...]
2023/12/29 07:58:32 [error] 58613#100259: *433021409 limiting requests, excess: *.203 by zone "", client: 115.227.85.250, server: , request: "GET some url"
Click to expand...

No idea, till we see packets (not?) being blocked.

Also possibly helpful:
# tcpdump -pen -i $interface
while it's happening.
 
I still get these lines in nginx logs from this ip address

Code: 
ipfw -t show
10201 646996458 796603072809 Mon Jan  8 17:17:05 2024 allow tcp from me to any established
10202 921723557 809587958518 Mon Jan  8 17:17:05 2024 allow tcp from me to any setup keep-state :default
10260       864        44441 Mon Jan  8 17:04:42 2024 deny log ip from table(1) to me
10261         8          608 Sat Jan  6 06:00:23 2024 deny ip from me to table(1)
10321         6          732 Mon Jan  8 03:01:08 2024 allow udp from y.y.y.y 53 to x.x.x.x via em
10327        11          691 Mon Jan  8 03:01:08 2024 allow udp from x.x.x.x to y.y.y.y 53 via em
10442       180         8100 Sat Jan  6 06:45:44 2024 allow tcp from any to me 80
10443       220         9900 Sun Jan  7 14:54:29 2024 allow tcp from any to me 443
65534    117489      5350330 Mon Jan  8 17:16:53 2024 deny log ip from any to any
65535      9438       754520 Sat Aug 19 23:27:34 2023 deny ip from any to any


Code: 
115.227.85.250 - - [08/Jan/2024:16:16:13 +0400] "GET some url HTTP/1.1" 200 6321 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [08/Jan/2024:16:16:13 +0400] "GET some url HTTP/1.1" 503 592 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [08/Jan/2024:16:16:14 +0400] "GET some url HTTP/1.1" 200 7207 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [08/Jan/2024:16:16:14 +0400] "GET some url HTTP/1.1" 503 592 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
 
VladiBG said:
move line 10260 and 10261 above any allow rules aka line 10201

Where's your check-state rule?
Click to expand...

If that's all the rules, state will be checked at the keep-state.

But that's for outbound TCP, and it follows allowing established traffic, which is weird.

This suggests that traffic from 115/8 is coming in response from outbound connections made by this system, no?
 
PMc said:
That's how it looks so far. One would need to look at ipfw -d show to be certain.
Click to expand...

Or ipfw -td show with timestamps - but you'd have to loop on that and test to catch a dynamic session with 115/8, amongst lots of traffic.

I think VladiBG is right: kill the unwanted traffic first, in and out, with logging if curious.

To me it seems better to use either stateless 'allow in setup' then 'allow established' and 'deny other' rules, OR inbound stateful 'setup keep-state' with an earlier check-state; I can't get my head around OP setup.
 
Code: 
# ipfw -td show
10260       55        2718 Wed Jan 10 11:49:41 2024 deny log ip from table(1) to me
10261        0           0                          deny ip from me to table(1)
10300        0           0                          check-state :default
10301 46616202 56736528926 Wed Jan 10 12:16:51 2024 allow tcp from me to any established
10302 66294438 57133598879 Wed Jan 10 12:16:51 2024 allow tcp from me to any setup keep-state :default
10421         6          732 Mon Jan  8 03:01:08 2024 allow udp from y.y.y.y 53 to x.x.x.x via em
10427        11          691 Mon Jan  8 03:01:08 2024 allow udp from x.x.x.x to y.y.y.y 53 via em
10548       38        7890 Wed Jan 10 01:49:05 2024 allow tcp from any to me 80
10549    77400     9946430 Wed Jan 10 12:16:50 2024 allow tcp from any to me 443
65534     6335      296969 Wed Jan 10 12:16:47 2024 deny log ip from any to any
65535     9438      754520 Sat Aug 19 23:27:41 2023 deny ip from any to any
## Dynamic rules (0):


Code: 
115.227.85.250 - - [10/Jan/2024:12:11:27 +0400] "GET some url HTTP/1.1" 503 592 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:27 +0400] "GET some url HTTP/1.1" 503 592 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:27 +0400] "GET some url HTTP/1.1" 503 592 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:28 +0400] "GET some url HTTP/1.1" 200 8538 "some url" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:28 +0400] "GET some url HTTP/1.1" 503 592 "some url" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:30 +0400] "GET some url HTTP/1.1" 200 13740 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:30 +0400] "GET some url HTTP/1.1" 200 1696 "some url" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:31 +0400] "GET some url HTTP/1.1" 200 8538 "some url" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
115.227.85.250 - - [10/Jan/2024:12:11:31 +0400] "GET some url HTTP/1.1" 200 8273 "some url" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

No lines in /var/log/security

Code: 
# cat security | grep 115.227.85.250
# bzcat security.* | grep 115.227.85.250
# bzcat security.1.* | grep 115.227.85.250
# bzcat security.0.* | grep 115.227.85.250
# bzcat security.2.* | grep 115.227.85.250
# bzcat security.3.* | grep 115.227.85.250
# bzcat security.4.* | grep 115.227.85.250
# bzcat security.5.* | grep 115.227.85.250
# bzcat security.6.* | grep 115.227.85.250
# bzcat security.7.* | grep 115.227.85.250
# bzcat security.8.* | grep 115.227.85.250
# bzcat security.9.* | grep 115.227.85.250
 
The keyword "me" match only the ip address which is configured on the interface. Instead use "any"

deny tcp from table(1) to any
 
VladiBG said:
The keyword "me" match only the ip address which is configured on the interface. Instead use "any"

deny tcp from table(1) to any
Click to expand...

Yes. I spent time looking at the latest, and wondered if the webserver was indeed on 'me', i.e. some interface on this system. Background config is lacking.

I still encourage adding 'log', at least untll no further mysteries.
 
Where's your web server (nginx) located. Is it inside Jail, VM, or at the same machine where IPFW is? Can you describe your network topology?
 
VladiBG said:
Where's your web server (nginx) located. Is it inside Jail, VM, or at the same machine where IPFW is? Can you describe your network topology?
Click to expand...
I gave you everything to answer my question. If no one knows the answer, tell me please how I can contact the developers of FreeBSD or IPFW.
Thank you.
 
Add this to your ipfw


ipfw add 1 deny ip from 115.0.0.0/8 to any
ipfw add 2 deny ip from any to 115.0.0.0/8


Then check the rule counters for matching packets.
 
akshin said:
I gave you everything to answer my question. If no one knows the answer, tell me please how I can contact the developers of FreeBSD or IPFW.
Thank you.
Click to expand...

If you take VladiBG's latest advice and deny any problem traffic first, that must remove the problem.

I would still add 'log' to those rules so you can see what's going on with that system.

[edit] and use 'any' rather than 'me' to remove ambiguity.
 
Hi again!

Code: 
115.114.12.122 - - [09/Mar/2024:13:00:48 +0400] "GET /some/url HTTP/1.1" 200 18218 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:01:12 +0400] "GET /some/url HTTP/1.1" 200 21995 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
115.114.12.122 - - [09/Mar/2024:13:01:21 +0400] "GET /some/url HTTP/1.1" 200 18222 "-" "Mozilla/5.0 (X11; CrOS x86_64 15359.58.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.134 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:01:52 +0400] "GET /some/url HTTP/1.1" 200 22003 "-" "Mozilla/5.0 (X11; CrOS x86_64 15359.58.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.134 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:03:07 +0400] "GET /some/url HTTP/1.1" 200 21359 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:03:10 +0400] "GET /some/url HTTP/1.1" 200 21354 "-" "Mozilla/5.0 (X11; CrOS x86_64 15359.58.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.134 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:05:31 +0400] "GET /some/url HTTP/1.1" 200 18207 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0"
115.114.12.122 - - [09/Mar/2024:13:05:31 +0400] "GET /some/url HTTP/1.1" 200 18361 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:06:02 +0400] "GET /some/url HTTP/1.1" 200 18205 "-" "Mozilla/5.0 (X11; CrOS x86_64 15359.58.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.134 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:06:07 +0400] "GET /some/url HTTP/1.1" 200 18372 "-" "Mozilla/5.0 (X11; CrOS x86_64 15359.58.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.134 Safari/537.36"
115.114.12.122 - - [09/Mar/2024:13:06:35 +0400] "GET /some/url HTTP/1.1" 200 18003 "-" "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"

Code: 
#!/bin/sh


cmd="ipfw -q add"
ipfw -q -f flush


# Block IP
ipfw add 1 deny ip from 115.0.0.0/8 to any
ipfw add 2 deny ip from any to 115.0.0.0/8


$cmd 100 check-state
$cmd 101 allow tcp from me to any established
$cmd 102 allow tcp from me to any setup keep-state
$cmd 103 allow ip from any to any via lo0
$cmd 104 deny ip from any to 127.0.0.0/8
$cmd 105 deny ip from 127.0.0.0/8 to any


# DDOS
$cmd 130 reject log tcp from any to any tcpflags fin, syn, rst, psh, ack, urg
$cmd 131 reject log tcp from any to any tcpflags !fin, !syn, !rst, !psh, !ack, !urg
$cmd 132 reject log tcp from any to any not established tcpflags fin
$cmd 133 deny log ip from any to any not verrevpath in

Code: 
# ipfw -td show
00001        39         1825 Sat Mar  9 00:44:16 2024 deny ip from 115.0.0.0/8 to any
00002         0            0                          deny ip from any to 115.0.0.0/8
00100         0            0                          check-state :default
00101 504288046 616085502474 Sat Mar  9 14:30:57 2024 allow tcp from me to any established
00102 729974921 646244781290 Sat Mar  9 14:30:57 2024 allow tcp from me to any setup keep-state :default
00103         0            0                          allow ip from any to any via lo0
00104         0            0                          deny ip from any to 127.0.0.0/8
00105         0            0                          deny ip from 127.0.0.0/8 to any
00130         0            0                          reject log tcp from any to any tcpflags syn,fin,ack,psh,rst,urg
00131         0            0                          reject log tcp from any to any tcpflags !syn,!fin,!ack,!psh,!rst,!urg
00132         0            0                          reject log tcp from any to any not established tcpflags fin
00133         0            0                          deny log ip from any to any not verrevpath in
 
Do you have some vpn interfaces which are not filtered by IPFW like OpenVPN, TOR, SSL VPN or some GRE tunnels which are activated after IPFW?
 
No vpn interfaces which are not filtered by IPFW like OpenVPN, TOR, SSL VPN or some GRE tunnels.
It is just production web server.
 
Maybe the IP address that you see in your NGINX is not the real ip address from which the request is recieved if it's forwarded from CDN or some other proxy.

NGINX access_log log the real client IP from X-Forwarded-For/X-Real-IP/cf-connecting-ip instead of the proxy IP

Have a nginx server receive requests from proxy or behind CDN, how to get client user's real IP instead of proxy/CDN servers IP in access log? This article provide two solutions based on `log_format` and nginx `realip` module.
djangocas.dev djangocas.dev

Can you make the following simple test to verify that IPFW is actually blocking.

Check your home public ip address. Then add it as rule number 3 and 4 for example:

If your home IP address is 1.2.3.4

ipfw add 3 deny ip from 1.2.3.4 to any 80
ipfw add 4 deny ip from 1.2.3.4 to any 443
Click to expand...

Then try to open your web site from your home and verify that you are unable to open it. And check the ipfw counters that rule 3,4 have hits. Then remove rule 3 and 4.
ipfw delete 3
ipfw delete 4
Click to expand...

Note:
You don't need to add it into your ipfw script file you can directly type the rules into the console without saving them.
 
VladiBG said:
Maybe the IP address that you see in your NGINX is not the real ip address from which the request is recieved if it's forwarded from CDN or some other proxy.
Click to expand...

There are a number of very strange aspects to this, which make me think we're not being shown the whole situation.

Given there's a flush command, it seems there have been over 600GB outbound TCP traffic since the latest ipfw restart, yet there's no rule allowing any inbound setup requests, stateful or otherwise?

Unless there are other rules not being shown - including default 65535 deny (or allow) other traffic rule - it makes no sense.

Then there's rule 1 last blocked 115/8 at 00:44, yet requests apparently served to 115.114.12.122 around [09/Mar/2024:13:06:35 +0400]

Is NGINX using the system timezone? From earlier, why were 'deny log' rules not being listed in /var/log/security? What version of FreeBSD?

This might make sense if OP was disabling ipfw from time to time - but how otherwise?
 
yes i don't see the entire rule set also. Initially i was thinking that during a reboot some connections sneak in but then the timer of last deny hit should be greater than log file of nginx.

Also i don't know why but on my server the ipfw internal iflist doesn't give any output.
 
VladiBG said:
yes i don't see the entire rule set also. Initially i was thinking that during a reboot some connections sneak in but then the timer of last deny hit should be greater than log file of nginx.

Also i don't know why but on my server the ipfw internal iflist doesn't give any output.
Click to expand...
The prohibiting rule comes first. And this rule should prohibit no matter what comes after this rule. If this is not the case, then the IPFW is not working correctly. Guys, IPFW doesn’t work and why don’t you want to admit it. I have already begun to doubt your integrity. Why do you require all the rules of IPFW You want to spy. Give me an email with the domain freebsd.org and I will email it to you.
Thank you.
 
You must log in or register to reply here.
Back
Top