PF PF doesn't load rules on boot?

F

forquare

Good evening all,

Yesterday (2nd December) I ran a freebsd-update(8) on my server, after a reboot the result was this:

Code: 
# freebsd-version -ku
14.3-RELEASE-p5
14.3-RELEASE-p6

After rebooting the server I set about updating my jails, as well as updating the packages in my jails. The package updates were taking ages and then failing, can't find the package server...
I then tried to access one of the websites hosted in a jail and couldn't connect. jexec(8) into a jail, it can't make DNS requests, cannot make http requests, cannot get out of the machine - it can communicate with other jails.

My connection to the server over SSH is fine.

pfctl -nf /etc/pf.conf runs without error and I have the following in my /etc/rc.conf:

Code: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
hostname="myhost"
keymap="uk.kbd"
ifconfig_em0="inet xxx.xxx.xxx.xxx netmask 0xffffffc0"

gateway_enable="YES"
defaultrouter="xxx.xxx.xxx.xxx"

sshd_enable="YES"
ntpd_enable="YES"
dumpdev="NO"
zfs_enable="YES"
bsdstats_enable="YES"
accounting_enable="YES"

daily_scrub_zfs_enable="YES"
daily_scrub_zfs_default_threshold="30"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

blacklistd_enable="NO"
blacklistd_flags="-r"

jail_enable=YES
cloned_interfaces="lo1"
ifconfig_lo1="10.0.0.1/24"

# Mail
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

zerotier_enable="YES"

vm_enable="YES"
vm_dir="zfs:zroot/bhyve"
vm_list="mc collabora-online"

node_exporter_enable="YES"
geom_exporter_enable="YES"
jail_exporter_enable="YES"
jail_exporter_listen_address="0.0.0.0:9452"
smartctl_exporter_enable="YES"
smartctl_exporter_listen_address="0.0.0.0:9633"

My /etc/pf.conf looks like this:
Code: 
ext_if = "em0"
vpn_zero_if="zt80s7s66sd8fg"
int_if = "lo1"
jail_net = $int_if:network
vm_if = "vm-public"
vm_net = $vm_if:network

jWEB = "10.0.0.2"
jWEB_TCP_PORTS = "{80, 443, 22222}"

jFORGEJO = "10.0.0.14"
jFORGEJO_TCP_PORTS = "{2222}"

vMINECRAFT = "10.1.1.5"
vMINECRAFT_TCP_PORTS = "{25565}"
vMINECRAFT_UDP_PORTS = "{25565}"

jSFTP = "10.0.0.12"
jSFTP_TCP_PORTS = "{2022}"

jPASSWORD = "10.0.0.8"
jPASSWORD_TCP_PORTS = "{3012}"

######################
######################
set skip on lo0
set skip on lo1
set skip on vm-public
scrub in on {$ext_if, $vpn_zero_if} all fragment reassemble
#####################

# Allow jails access to outside world
nat on $ext_if from $jail_net to any -> ($ext_if)

# Allow jails to access VPN
nat on $vpn_zero_if from $jail_net to any -> ($vpn_zero_if)

# Allow vm-bhyve access to the outside world
nat on $ext_if from {$vm_net} to any -> ($ext_if)

# Redirect web traffic to web jail
rdr pass on {$ext_if, $vpn_zero_if} inet proto tcp to port $jWEB_TCP_PORTS -> $jWEB

# Redirect Git traffic to Git jail
rdr pass on {$vpn_zero_if} inet proto tcp to port $jFORGEJO_TCP_PORTS -> $jFORGEJO

# Redirect Minecraft traffic to MC VM
rdr pass on $ext_if inet proto tcp to port $vMINECRAFT_TCP_PORTS -> $vMINECRAFT
rdr pass on $ext_if inet proto udp to port $vMINECRAFT_UDP_PORTS -> $vMINECRAFT

# Redirect SFTP traffic to SFTP jail
rdr pass on {$ext_if, $vpn_zero_if} inet proto tcp to port $jSFTP_TCP_PORTS -> $jSFTP

# Redirect PASSWORD traffic to PASSWORD jail
rdr pass on {$vpn_zero_if} inet proto tcp to port $jPASSWORD_TCP_PORTS -> $jPASSWORD


block in all
pass out all keep state
antispoof for {$ext_if, $vpn_zero_if} inet


# Allow the jail and VM traffic to be translated
pass from { $jail_net, $vm_net } to any keep state

# Only allow SSH on VPN
pass in quick on $vpn_zero_if proto tcp from any to $vpn_zero_if port 22

# Only allow prometheus on VPN
pass in quick on $vpn_zero_if proto tcp from any to $vpn_zero_if port {9100, 9248, 9633, 9452}

# Allow VPN traffic to NATed jails
pass in quick on $vpn_zero_if inet from any to $jail_net keep state

# Allow pinging on VPN
pass in quick on {$vpn_zero_if} inet proto icmp all icmp-type echoreq

This was all working prior to running the last update command.

Now, if I run service pf reload or pfctl -f /etc/pf.conf once the box is up and running, everything works just fine!
If I reboot the server, I'm back to having the rules not load again...

This is affecting two other FreeBSD servers I run with the same version.

Any pointers would be most appreciated!
Ben
 
I think it's something related with the interfaces that are created after pf loads the rules file. Since they don't exist, an error will occur. Use parentheses so that pf resolves the interface to the IP address at runtime, but for those macros that use :network, it is preferable to hardcode those values.
 
DtxdF said:
I think it's something related with the interfaces that are created after pf loads the rules file. Since they don't exist, an error will occur. Use parentheses so that pf resolves the interface to the IP address at runtime, but for those macros that use :network, it is preferable to hardcode those values.
Click to expand...
I hope I've understood you well enough - I've made two changes:

Firstly, I've moved my PF block in /etc/rc.conf to the bottom - not sure if order matters here, but it was previously above the creation of the lo1 interface and starting the VM stuff.

Secondly, I've changed the top of /etc/pf.conf to this:

Code: 
ext_if = "em0"
vpn_zero_if="zt80lm2s8e3rlsd"
#int_if = "lo1"
jail_net = "10.0.0.0/24"
#vm_if = "vm-public"
vm_net = "10.1.1.0/24"

Rebooted and still getting the same issue of no rules loading. I then removed the other references to lo1 and vm-public from /etc/pf.conf and rebooted yielding the same issue.

Just also want to reiterate that this setup has been working largely untouched for around six years, the server has been upgraded from 12.x-RELEASE to 13.x-RELEASE, to 14.x-RELEASE - it seems the latest patch is where it started failing.
 
forquare said:
Firstly, I've moved my PF block in /etc/rc.conf to the bottom - not sure if order matters here
Click to expand...
Nope, order is irrelevant, rc.conf contains variable assignments. Unless you try to assign a value multiple times to the same variable, the order of the assignments is not relevant.
 
SirDice said:
Nope, order is irrelevant, rc.conf contains variable assignments. Unless you try to assign a value multiple times to the same variable, the order of the assignments is not relevant.
Click to expand...
Thanks SirDice - I expected as such but in some late night searching I found a few old posts on various websites that suggested order might matter.
Also, thank you for moving to the Firewalls forum - when I posted late last night I overlooked it.
 
You must log in or register to reply here.
Back
Top