Solved PF block not stopping access to my jail

[Rand0m]

Hello there,

I have a jail inside a VM. I installed Gitea inside the Jail and configured PF (nat) to forward traffic coming on port 2000 to the jail port 3000 (The gitea web application) and left port 10000 for the ssh (for git) inside the jail.

All is okay so far till recently I checked my sshd logs and found that some IPs repeatedly trying to login -although I configured ssh to use keys only I thought best course of action is to collected those IPs and block them.
For this I add them to /etc/badips file and added table in my PF.conf and blocked all traffic from or to those IPs.
For making sure it is working, I added my work IP to the /etc/badips file and tested the connectivity to the gitea webapp inside the jail. I was expecting not to be able to log to the page, yet I am still to use the webapp.

My pf.conf (I know it is not the most secure configuration but it is work in progress)

#Define the interfaces
ext_if = "vtnet0"
int_if = "lo1"
jail_net = $int_if:network

#Define the IP address of jail and the ports

git_jail = "192.168.1.1"
git_jail_http_port = "2000"
git_ext_http_port = "3000"
git_jail_ssh_port = "10000"
git_jail_TCP_PORTS = "{ 10000, 3000 }"

#Define the NAT for the jails
nat on $ext_if from $jail_net to any -> ($ext_if)

# Redirect traffic on port 3000 to the git jail
#rdr pass on $ext_if inet proto tcp to port $git_jail_TCP_PORTS -> $git_jail
rdr pass on $ext_if inet proto tcp from any to any port $git_ext_http_port  -> $git_jail port $git_jail_http_port
rdr pass on $ext_if inet proto tcp from any to any port $git_jail_ssh_port  -> $git_jail port $git_jail_ssh_port

table <badhosts> persist file "/etc/badips"
block in quick from <badhosts>
block out quick to <badhosts>

Sample of the content of /etc/badips file

 % sudo cat /etc/badips
103.79.141.190
104.163.168.159
111.13.138.175
112.65.130.130
112.65.140.133
113.193.0.206

I tried to display the ips and check on my work ip if it is there
pfctl -t badhosts -T show
And I can see it

# cat /etc/badips | xargs pfctl -t badhosts -Tt
95/95 addresses match.

I must have something wrong, but couldn't figure it out.
I tried using block on $ext_if and other combination I found on the internet nothing worked

Thanks a lot

 

[sko]

If you are already connected to the host via the IP you are adding to the blacklist, PF still has it in its state-tables and will not re-evaluate rules for that ip/port combination. Kill all states from/to the IP with:
pfctl -k $ip
Of course this will also kill your ssh connection!
For testing use pfctl to add the ip to the table or use a different config file, loaded via pfctl -f /not/etc/pf.conf. Then set up a cronjob that reloads pf with a known-working ruleset (and tables) so you can access your server if you screw things up. This is BTW always a good idea when configuring PF on a remote host.


Regarding the background noise on your sshd (and probably other services/ports): just use overload rules to add bruteforce attackers to a blacklist - this also works for other services/ports than ssh and does not need manual intervention (i.e. manually adding addresses to a file)

 

[Rand0m]

Hi Sko,
Thanks for your reply, I was accessing the VM at the time of testing from the web portal so not via SSH, and I did later another test ssh from workstation and adding my Mobile 4G IP to badips (which has different IP) and refreshing the "pfctl -f...". Still was able to access the web portal from my phone.

Is my pf.conf configuration correct?

I still want to see this working, but as for

Regarding the background noise on your sshd (and probably other services/ports): just use overload rules to add bruteforce attackers to a blacklist - this also works for other services/ports than ssh and does not need manual intervention (i.e. manually adding addresses to a file)

Would you please elaboratore more.
Thanks again

 

[sko]

I'd try with these rules:

rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_ext_http_port -> $git_jail port $git_jail_http_port
rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_jail_ssh_port -> $git_jail port $git_jail_ssh_port

This notation also makes it clearer when reading the rules that <badhosts> are blocked from access. You don't have to read all the way to the "block" rules further down the file.

Also I position rules for blacklists as "block quick" at the beginning of the "rules"-section, so blacklisted hosts are blocked early on and PF doesn't evaluate all other rules before finally blocking the connection. (PF parses rules as "last match wins")

Would you please elaboratore more

You can tell PF how many connections from one IP are allowed and at what rate:

pass proto tcp from any to $ext_if port $tcp_services (max-src-conn 10, max-src-conn-rate 10/5, overload <overloaders> flush global)

This translates to:
- max 10 simultaneous active connections
- max 10 connection attempts in 5 seconds
If an IP triggers one of these limits, it will be added to the <overloaders> table


Another (additional) way to get rid of the constant login attempts from bots is sshguard(8) (package: sshguard-pf). By default it monitors /var/log/auth.log and /var/log/maillog, assigns scores to IPs that generate log entries and adds them to the <sshguard> PF table.

 

[Rand0m]

Thanks

I'd try with these rules:

rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_ext_http_port -> $git_jail port $git_jail_http_port
rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_jail_ssh_port -> $git_jail port $git_jail_ssh_port

This notation also makes it clearer when reading the rules that <badhosts> are blocked from access. You don't have to read all the way to the "block" rules further down the file.

Also I position rules for blacklists as "block quick" at the beginning

will denfinitely do that,

Sorry to ask again, but this block them to the jail only which is great, yet how to make the block statement block them to host as well. It is good to learn what is wrong with my configuration.

I will add the other configurations and sshguard , thanks a lot for that

 

[Rand0m]

Well I moved the block statements to the top (beneath the nat) and as soon as I uncomment any of those block statements I get an error straight away

#block in quick from <badhosts>
#block out quick to <badhosts>
#block in quick on $ext_if from <badhosts> to any
#block quick from <badhosts>

(tried them all)


# pfctl -F all -f /etc/pf.conf
rules cleared
nat cleared
0 tables deleted.
0 states cleared
source tracking entries cleared
pf: statistics cleared
pf: interface flags reset
/etc/pf.conf:29: Rules must be in order: options, normalization, queueing, translation, filtering
/etc/pf.conf:30: Rules must be in order: options, normalization, queueing, translation, filtering
pfctl: Syntax error in config file: pf rules not loaded


the below change worked fine. but this is great for the jail but it doesn't block the badhosts from trying to access VM host itself.

rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_ext_http_port -> $git_jail port $git_jail_http_port
rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_jail_ssh_port -> $git_jail port $git_jail_ssh_port

and on different note, the overloads is been added no issues, not sure if the block works or will suffer the same issue as the others

table <bruteforce> persist

block quick from <bruteforce>
pass inet proto tcp from any to $ext_if port $tcp_services flags  S/SA keep state (max-src-conn 10, max-src-conn-rate 10/5, overload <bruteforce> flush global)

 

[SirDice]

Looks like you moved the block rules a bit too far up. The order is:

nat
rdr
block/pass

So make sure your block rules are right after the last redirects.

 

[Rand0m]

Looks like you moved the block rules a bit too far up. The order is:

nat
rdr
block/pass

So make sure your block rules are right after the last redirects.

Hi SirDice
as you can see above in the first thread, the blocks where below the rdr but didn't work.

 

[sko]

Did you left the table definition for <badhosts> at the bottom? Tables have to be defined before they are used.

It's always best to group macros and table definitions at the top and not scatter them around between the rules.

 

[Rand0m]

This is my current pf.conf, if I replace !<badhosts> with any in the rdr I will still be able connect to the web app at port 3000 from my phone although it is in the badips file and consequently badhosts table.

#Define the interfaces
ext_if = "vtnet0"
int_if = "lo1"
jail_net = $int_if:network
tcp_services = "{ ssh, smtp, domain, www, pop3, auth, pop3s }"

table <badhosts> persist file "/etc/badips"
table <bruteforce> persist

#Define the IP address of jails
#as well as port to be allowed redirected

git_jail = "192.168.1.1"
git_jail_http_port = "2000"
git_ext_http_port = "3000"
git_jail_ssh_port = "10000"
git_jail_TCP_PORTS = "{ 10000, 3000 }"


#Define the NAT for the jails
nat on $ext_if from $jail_net to any -> ($ext_if)

# Redirect traffic on port 3000 to the git jail
rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_ext_http_port  -> $git_jail port $git_jail_http_port
rdr pass on $ext_if inet proto tcp from !<badhosts> to any port $git_jail_ssh_port  -> $git_jail port $git_jail_ssh_port

block quick from <badhosts> 

block quick from <bruteforce>
pass inet proto tcp from any to $ext_if port $tcp_services flags  S/SA keep state (max-src-conn 10, max-src-conn-rate 10/5, overload <bruteforce> flush global)

 

[sko]

This is intended - when redirecting a packet, it originates from the interface and port it has been redirected to. Translation/redirection is processed before other rules (this is why you had to move the rdr rules in front of the block/pass rules to stop pfctl from complaining), your block rules for <badhosts> only apply for $ext_if, but the packets have already been redirected (they should now originate from the loopback device)!
The pf.conf(5) man page goes into some detail on translation/redirection and specifically mentions this behaviour.
This man page is BTW always a good starting point when PF doesn't seem to behave as expected.

To have the block rule match packets after redirection, have to remove the interface from the rule. In fact all of your 3 somewhat redundant "block" rules can be combined and shortened to this:

block quick from <badhosts> to any

However, you'd still waste CPU time for the redirection. So the more efficient approach is the "from !<badhosts>" restriction for the rdr rule.

A completely different approach would be to attach each jail to its own loopback device and address (127.0.0.[2...254]), so you can apply rules specifically to these devices/IPs. On some configurations this might be cleaner and easier to read/understand/debug at 3 in the morning when something went wrong....

 

[Rand0m]

This is intended - when redirecting a packet, it originates from the interface and port it has been redirected to. Translation/redirection is processed before other rules (this is why you had to move the rdr rules in front of the block/pass rules to stop pfctl from complaining), your block rules for <badhosts> only apply for $ext_if, but the packets have already been redirected (they should now originate from the loopback device)!
The pf.conf(5) man page goes into some detail on translation/redirection and specifically mentions this behaviour.
This man page is BTW always a good starting point when PF doesn't seem to behave as expected.

To have the block rule match packets after redirection, have to remove the interface from the rule. In fact all of your 3 somewhat redundant "block" rules can be combined and shortened to this:

block quick from <badhosts> to any

However, you'd still waste CPU time for the redirection. So the more efficient approach is the "from !<badhosts>" restriction for the rdr rule.

A completely different approach would be to attach each jail to its own loopback device and address (127.0.0.[2...254]), so you can apply rules specifically to these devices/IPs. On some configurations this might be cleaner and easier to read/understand/debug at 3 in the morning when something went wrong....

Thanks so much for all the help, highly appreciated

 

[SirDice]

your block rules for <badhosts> only apply for $ext_if, but the packets have already been redirected (they should now originate from the loopback device)!

No, a rdr is a destination NAT. Which means the destination address (and optionally the destination port) is rewritten. The source address and port of the packet is not touched. So for a block or pass rule you need to use the original source address and the redirected destination address.

A nat rule is a source NAT. Here the source address (and optionally the source port) is rewritten, the destination address and port is not changed.

But, one of the things that still trip me up with PF is the fact that you have two interfaces to deal with. Traffic comes in on $ext_if and goes out $int_if. So for traffic passing through the firewall you need 2 sets of rules, one set for the traffic coming in on $ext_if and another set for the (same) traffic going out $int_if

 

[sko]

No, a rdr is a destination NAT. Which means the destination address (and optionally the destination port) is rewritten. The source address and port of the packet is not touched. So for a block or pass rule you need to use the original source address and the redirected destination address.

A nat rule is a source NAT. Here the source address (and optionally the source port) is rewritten, the destination address and port is not changed.

Thanks for clearing that up and sorry for the mistake - I was looking for and tcpdump-ing a rdr rule here (I use them very seldomly and even less with jails...) and completely missed that the rdr I found and observed directs to an interface (tun) on which NAT also applies, hence the rewrite of the source. My bad - sorry for the confusion.
The manpage is - of course - also correct about the rewrite of the destination vs source on rdr and nat and how this affects filters.

Just out of curiosity: with the rdr rule excluding <badhosts> and the generalized block from <badhosts> to any the ruleset should work as intended. blueCub, could you comment on that?

But, one of the things that still trip me up with PF is the fact that you have two interfaces to deal with. Traffic comes in on $ext_if and goes out $int_if. So for traffic passing through the firewall you need 2 sets of rules, one set for the traffic coming in on $ext_if and another set for the (same) traffic going out $int_if

For more complex scenarios/configurations I think this is beneficial, as the ruleset can be much more fine-grained yet still somewhat readable and even portable (I'm using ~70% of our ruleset on all our gateways - site-specific configuration is added via include-files from within the generalized pf.conf). But of course it can add a lot of (unnecessary) complexity on more basic configurations and you have to set some guidelines and stick to them.
I always try to match packets as they enter the system, especially for block/pass rules and even try to avoid the need for specific in/out distinction. Mixing in/out rules throughout the whole config can mess up a ruleset beyond any recognition, making it practically impossible to grasp or even debug the path of a packet.

I had to deal with linux/iptables based firewalls before PF - the complexity and abstraction with iptables is just on another level compared to PF, even on much less complex networks. Debugging a firewall with a ruleset grown with the network over ~5 years is a complete nightmare of itself, but additionally having to deal with the braindead iptables-syntax and its weird behavior in many common cases (bridges breaking NAT...) is just pure madness.

 

[Rand0m]

Just out of curiosity: with the rdr rule excluding <badhosts> and the generalized block from <badhosts> to any the ruleset should work as intended. blueCub, could you comment on that?

Thanks sko for the support, I tested it again and now both redirect to the jail and access to host is blocked for <badhosts>