PF SSHGuard behaving strangely with PF

J

jokibear

SSHGuard in combination with pf on FreeBSD 14.0-RELEASE (fresh vanilla installation on Hetzner VPS) is behaving strangely. After starting, it either terminates immediately or after some variable time, indicating a potential issue with pf. Below are the details:

System Info:
Code: 
FreeBSD myserver 14.0-RELEASE-p6 FreeBSD 14.0-RELEASE-p6 #0: Tue Mar 26 20:37:02 UTC 2024 root@arm64-builder.daemonology.net:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64

1. Apparent Undeterministic Behaviour

1.1 First Start Attempt - terminates immediately
Code: 
root@myserver:~ # export SSHGUARD_DEBUG=1
root@myserver:~ # sshguard -a 30 -i /var/run/sshguard.pid
pfctl: Unknown error: -1.
sshg-fw-pf: Could not initialize firewall

1.2 Second Start Attempt - does not terminate immediately
Code: 
root@myserver:~ # sshguard -a 30 -i /var/run/sshguard.pid
pfctl: Unknown error: -1.
sshg-fw-pf: Could not initialize firewall
sshguard 2779 - - whitelist: add IPv4 block: 127.0.0.0 with mask 8.
sshguard 2779 - - whitelist: add '::1' as plain IPv6.
sshguard 2779 - - Now monitoring attacks.
^Csshguard 2779 - - Exiting on signal.
Terminated

I terminated this intentionally, just to see, how the 3rd call will behave.

1.3 Third Start Attempt - does not terminate immediately
Code: 
root@myserver:~ # date
Mon Apr 15 15:05:09 CEST 2024
root@myserver:~ # sshguard -a 30 -i /var/run/sshguard.pid
pfctl: Unknown error: -1.
sshguard 2801 - - whitelist: add IPv4 block: 127.0.0.0 with mask 8.
sshguard 2801 - - whitelist: add '::1' as plain IPv6.
sshguard 2801 - - Now monitoring attacks.
sshg-fw-pf: Could not initialize firewall
sshguard 2801 - - Purging old attackers.
sshguard 2801 - - Attack from "90.156.57.221" on service SSH with danger 10.
sshguard 2801 - - Purging old attackers.
sshguard 2801 - - Attack from "90.156.57.221" on service SSH with danger 2.
sshguard 2801 - - Purging old attackers.
sshguard 2801 - - Attack from "176.74.65.123" on service SSH with danger 10.
sshguard 2801 - - Purging old attackers.
sshguard 2801 - - Attack from "176.74.65.123" on service SSH with danger 2.
root@myserver:~ #
oot@myserver:~ # date
Mon Apr 15 15:08:33 CEST 2024

It just silently terminates without any log info, also /var/log/auth.log or /var/log/messages show nothing informative here. SSHGuard just stops logging.


2. Checking PF for SSHGuard
Code: 
root@myserver:~ # pfctl -T show -t sshguard
pfctl: Unknown error: -1.
root@myserver:~ # service sshguard status
sshguard is running as pid 2605.
root@myserver:~ # service pf status
Status: Enabled for 0 days 01:42:46 Debug: Urgent

State Table Total Rate
  current entries 5
  searches 96955 15.7/s
  inserts 1446 0.2/s
  removals 1441 0.2/s
Counters
  match 4111 0.7/s
  bad-offset 0 0.0/s
  fragment 0 0.0/s
  short 0 0.0/s
  normalize 3 0.0/s
  memory 0 0.0/s
  bad-timestamp 0 0.0/s
  congestion 0 0.0/s
  ip-option 0 0.0/s
  proto-cksum 0 0.0/s
  state-mismatch 0 0.0/s
  state-insert 0 0.0/s
  state-limit 0 0.0/s
  src-limit 0 0.0/s
  synproxy 0 0.0/s
  map-failed 0 0.0/s


3. PS Aux after starting SSHGuard
Code: 
USER    PID %CPU %MEM   VSZ   RSS TT  STAT STARTED      TIME COMMAND
root      0  0.0  0.0     0  1136  -  DLs  12:46     0:01.15 [kernel]
root      1  0.0  0.0 11640  1260  -  SLs  12:46     0:00.04 /sbin/init
root      2  0.0  0.0     0    32  -  WL   12:46     0:01.12 [clock]
root      3  0.0  0.0     0    48  -  DL   12:46     0:00.00 [crypto]
root      4  0.0  0.0     0    48  -  DL   12:46     0:00.22 [cam]
root      5  0.0  0.0     0   688  -  DL   12:46     0:00.91 [zfskern]
root      6  0.0  0.0     0    16  -  DL   12:46     0:00.66 [rand_harvestq]
root      7  0.0  0.0     0    48  -  DL   12:46     0:00.80 [pagedaemon]
root      8  0.0  0.0     0    16  -  DL   12:46     0:00.00 [vmdaemon]
root      9  0.0  0.0     0    48  -  DL   12:46     0:00.14 [bufdaemon]
root     10  0.0  0.0     0    16  -  DL   12:46     0:00.00 [audit]
root     11 200.0  0.0     0    32  -  RNL  12:46   255:59.34 [idle]
root     12  0.0  0.0     0   304  -  WL   12:46     0:03.47 [intr]
root     13  0.0  0.0     0    48  -  DL   12:46     0:00.00 [geom]
root     14  0.0  0.0     0    16  -  DL   12:46     0:00.00 [sequencer 00]
root     15  0.0  0.0     0    80  -  DL   12:46     0:00.20 [usb]
root     16  0.0  0.0     0    16  -  DL   12:46     0:00.04 [vnlru]
root     17  0.0  0.0     0    16  -  DL   12:46     0:00.06 [syncer]
root     18  0.0  0.0     0    16  -  DL   12:46     0:00.00 [busdma]
root    726  0.0  0.1 13192  2540  -  Is   12:46     0:00.00 dhclient: system.syslog (dhclient)
root    729  0.0  0.1 13192  2644  -  Is   12:46     0:00.00 dhclient: vtnet0 [priv] (dhclient)
_dhcp   794  0.0  0.1 13196  2700  -  ICs  12:46     0:00.00 dhclient: vtnet0 (dhclient)
root    795  0.0  0.1 14504  3836  -  Ss   12:46     0:00.50 /sbin/devd
root    859  0.0  0.0     0    16  -  DL   12:46     0:02.07 [pf purge]
root   1007  0.0  0.1 12896  2676  -  Ss   12:46     0:00.39 /usr/sbin/syslogd -s
ntpd   1063  0.0  0.2 23020  7408  -  Ss   12:46     0:00.47 /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift
root   1107  0.0  0.2 22460  9540  -  Is   12:46     0:00.40 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
root   1110  0.0  0.1 12940  2468  -  Is   12:46     0:00.03 /usr/sbin/cron -s
root   1165  0.0  0.2 22696 10084  -  Is   12:46     0:00.03 sshd: myuser [priv] (sshd)
myuser  1168  0.0  0.3 22696 10372  -  S    12:46     0:04.39 sshd: myuser@pts/0 (sshd)
root   2678  0.0  0.1 13392  2832  -  Ss   14:54     0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root   2679  0.0  0.1 13028  2492  -  SC   14:54     0:00.00 tail -F -n 0 /var/log/auth.log /var/log/maillog
root   2680  0.0  0.1 17640  5236  -  SC   14:54     0:00.00 /usr/local/libexec/sshg-parser
root   2681  0.0  0.1 13420  2816  -  SC   14:54     0:00.00 /usr/local/libexec/sshg-blocker -a 30 -p 120 -s 1800 -N 128 -n 32
root   2689  0.0  0.1 17640  5224  -  Ss   14:54     0:00.00 sshg-parser: system.net (sshg-parser)
root   2691  0.0  0.1 13420  2776  -  Ss   14:54     0:00.00 sshg-blocker: system.net (sshg-blocker)
root   2694  0.0  0.1 13028  2388  -  S    14:54     0:00.00 tail: system.fileargs (tail)
root   2695  0.0  0.1 13516  2844  0  R+   14:54     0:00.00 ps aux


4. Config Files
/etc/rc.conf:
Code: 
hostname="myserver"
keymap="uk.iso.kbd"
ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
moused_nondefault_enable="NO"
dumpdev="AUTO"
zfs_enable="YES"
pf_enable="YES"
sshguard_enable="YES"

/etc/pf.conf:
Code: 
ext_if="vtnet0"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
table <sshguard> persist
block in all
pass out quick keep state
antispoof for $ext_if inet
block in quick from <sshguard>
pass in inet proto tcp from any to any port ssh flags S/SA keep state
pass in on $ext_if from any to any

/usr/local/etc/sshguard.conf:
Code: 
#!/bin/sh
# sshguard.conf -- SSHGuard configuration

# Options that are uncommented in this example are set to their default
# values. Options without defaults are commented out.

#### REQUIRED CONFIGURATION ####
# Full path to backend executable (required, no default)
#BACKEND="/usr/local/libexec/sshg-fw-hosts"
#BACKEND="/usr/local/libexec/sshg-fw-ipfw"
BACKEND="/usr/local/libexec/sshg-fw-pf"

# Space-separated list of log files to monitor. (optional, no default)
FILES="/var/log/auth.log /var/log/maillog"

# Shell command that provides logs on standard output. (optional, no default)
# Example 1: ssh and sendmail from systemd journal:
#LOGREADER="LANG=C /usr/bin/journalctl -afb -p info -n1 -t sshd -t sendmail -o cat"
# Example 2: ssh from os_log (macOS 10.12+)
#LOGREADER="/usr/bin/log stream --style syslog --predicate '(processImagePath contains \"sshd\")'"

#### OPTIONS ####
# Block attackers when their cumulative attack score exceeds THRESHOLD.
# Most attacks have a score of 10. (optional, default 30)
THRESHOLD=30

# Block attackers for initially BLOCK_TIME seconds after exceeding THRESHOLD.
# Subsequent blocks increase by a factor of 1.5. (optional, default 120)
BLOCK_TIME=120

# Remember potential attackers for up to DETECTION_TIME seconds before
# resetting their score. (optional, default 1800)
DETECTION_TIME=1800

# Size of IPv6 'subnet to block. Defaults to a single address, CIDR notation. (optional, default to 128)
IPV6_SUBNET=128

# Size of IPv4 subnet to block. Defaults to a single address, CIDR notation. (optional, default to 32)
IPV4_SUBNET=32

#### EXTRAS ####
# Full path to PID file (optional, no default)
#PID_FILE=/var/run/sshguard.pid

# Colon-separated blacklist threshold and full path to blacklist file.
# (optional, no default)
#BLACKLIST_FILE=120:/var/db/sshguard/blacklist.db

# IP addresses listed in the WHITELIST_FILE are considered to be
# friendlies and will never be blocked.
#WHITELIST_FILE=/usr/local/etc/sshguard.whitelist

# If PARSER is unset, SSHGuard will use the installed sshg-parser as its
# parser. Setting PARSER overrides this, so that you can use your own parser.
#PARSER=

# Run POST_PARSER as a filter after the parser. POST_PARSER must read as input
# and produce as output lines in the format used by sshg-parser. This example
# implements primitive whitelisting, preventing sshg-blocker from seeing
# attacks from 1.2.3.4. Unlike whitelisting, attacks filtered by POST_PARSER
# are not logged by SSHGuard.
#POST_PARSER="grep -v 1.2.3.4"

Again, the files /var/log/auth.log or /var/log/messages show nothing. SSHGuard just stops producing messages, but no error.

I wonder what is wrong with the setup. I used SSHGuard with previous versions of FreeBSD, always worked flawlessly. But now I'm biting on granite.
 
I have sshguard with PF running fine on 14.0, so it's not a common thing. Did you perhaps enable a bunch of those security settings with the installation?
 
No, nothing. Did a reinstall on a amd64 VPS, was arm64 before. Exactly the same issue. Maybe it's a Hetzner-related problem. On the other hand, in both cases it was installed from the official Freebsd ISO - hence vanilla kernel. Will try to use IPFW (which I don't like so much) and another VPS provider with pf. - I have absolutely no clue what is going on.
 
After the system has been rebooted, does pfctl -sr work correctly? You haven't set the security level to 3, did you?

Code: 
     3     Network secure mode - same as highly secure mode, plus IP packet
           filter rules (see ipfw(8), ipfirewall(4) and pfctl(8)) cannot be
           changed and dummynet(4) or pf(4) configuration cannot be adjusted.
 
Code: 
sysctl kern.securelevel
kern.securelevel: -1



pfcrl -sr somehow does not reflect the sshguard part of /etc/pf.conf

pfctl -sr
Code: 
scrub in on vtnet0 all fragment reassemble
block return in all
pass out quick all flags S/SA keep state
block drop in on ! vtnet0 inet from 167.235.72.123 to any
block drop in inet from 167.235.72.123 to any
pass in inet proto tcp from any to any port = ssh flags S/SA keep state

/etc/pf.conf:

Code: 
ext_if="vtnet0"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
# SSHGuard integration
table <sshguard> persist
block in all
pass out quick keep state
antispoof for $ext_if inet
# Pass in for SSH and check for SSHGuard blocks
block in quick from <sshguard>
pass in inet proto tcp from any to any port ssh flags S/SA keep state
pass in on $ext_if from any to any
 
jokibear said:
pfcrl -sr somehow does not reflect the sshguard part of /etc/pf.conf
Click to expand...
It won't show the table definition, but it should show that block rule.

Code: 
root@maelcum:~ # pfctl -sr
scrub in on ng0 all fragment reassemble
block drop in quick on ng0 from <crap> to any
block drop in log quick on ng0 proto tcp from <sshguard> to any label "Brute-Force"
block drop in log quick on ng0 inet6 proto tcp from <sshguard> to any label "Brute-Force"
<bunch more rules>
 
Hi,

I did some test on a fresh VM(14.0-RELEASE), as I never tried security/sshguard before it was a good opportunity for testing it.
I used the same rules you have in your /etc/pf.conf and I didn't face any trouble.
It works normally, attacks are detected and blocked immediately.
Logs /var/log/messages /var/log/auth.log show the relevant information as intended and pfctl shows them in the table.

Code: 
# Show what is happening in live with the relevant pf rule number (useful to understand which rule is involved)
tcpdump -e -i pflog0 port 22
# Identify what is the rule #4 (for example) that showed up in the previous tcpdump command
pfctl -gsr | grep "@4"
# Show all tables's information
pfctl -vvsTables
# Show what is in the table sshguard (only if not empty)
pfctl -t sshguard -T show

Since I didn't notice anything wrong, I looked again at your log and noticed this pfctl: Unknown error: -1 comes multiple time, and it may be what your problem is.
A possible increase of net.pf.request_maxcount could solve your problem, the default is 65535, I can't tell which value you should put instead as I am not qualify enough for that so I let you decide.

The following thread is where I found what I explained above:
forums.freebsd.org

Solved - update to FreeBSD-13.2 and pf now throws unknown error -1

pfctl -t BLOCKPERM -T replace -f /var/db/pf/pf_block_perm pfctl: Unknown error: -1. tail /var/db/pf/pf_block_perm . . . 223.240.83.206 223.241.100.90 223.241.222.151 223.242.38.53 223.243.101.149 223.244.87.96 223.245.0.5 223.255.146.2 223.255.146.94 223.255.187.154 Q. What is causing this...
forums.freebsd.org forums.freebsd.org

If this solve the issue could this mean that the server is heavily attacked, so pf has a lot of requests to deal with? or just the value is not well adjusted in some case?
 
You must log in or register to reply here.
Back
Top