Firewall Malfunction = Me Confused

Ruler2112

Ruler2112

I have a mail server that's been up for a couple years running FreeBSD 7.1. I've always used pf as the firewall and have generally found it to be very reliable and easy to use. However, I had a problem yesterday that is twisting my brain into a knot.

About noon EDT, a user came to me and told me that he wasn't getting sales leads from one source. He also discovered that he was unable to send mail to anybody at my domains from his gmail account. I checked and there was nothing in the server logs about any rejected messages, but I did verify by both using his gmail account and my own fastem account to send mail to myself at work - nothing arrived and nothing appeared in the logs. I did discover that I'd get the dreaded
Code: 
sendto: Operation not permitted
when trying to ping any of the affected machines. (It was weird - I could ping certain IPs and others would produce the above error.) Even openntpd started spewing out similar errors to the console. I assumed that these were related and started troubleshooting.

I tried restarting postfix, amamisd-new, and postgrey. Nothing. (This is when I found I couldn't ping.) I restarted the firewall with [cmd=]pfctl -d;pfctl -e -f /etc/pf.rules[/cmd] - nada. Then I restarted ppp to refresh my PPPoE link (DSL), which also restarts the firewall - still had the same symptoms. Likewise after rebooting the server. I found a reference online that suggested it might be caused by a backbone outage, so I called my ISP. Doug hooked me up with a different user ID/password that put me on a completely different subnet - obviously I couldn't receive mail because my static IP was tied to my regular user ID, but the ping issues remained. After some more troubleshooting, Doug suggested that it could be a firewall causing the problem. I didn't think this was the case, but enabled my old rule-set anyway just to prove him wrong. Imagine my surprise when things started working! I left the old rules (without traffic shaping or block tables) in place. The PPPoE link re-enabled my current rule-set automatically when it re-authenticated earlier today - there have been no problems since. I just tested and I can send mail from my home account no problem.

I've been trying to figure out what happened, why, and a way to fix it since, but it seems to defy logical explanation. I checked the current and old block tables (I keep files from the past week) and haven't found any of the IPs this happened with in there. I thought that one or more of the in-memory block tables may have gotten corrupt, but restarting the firewall, PPP link, or server should have cleared the problem if that were the case. If it were a problem with the current rule-set itself, why did it not behave that way for the ~week I was gone on vacation immediate prior to this and why is it not doing the same thing now???

It also affected some machines and not others. Most of the google addresses I tried resulted in errors, but only about half of the yahoo addresses had trouble - the other half went through fine. I could ping another of my company's public servers over the internet, but I could not ping my ISPs main web server. People were getting mail, but only from certain sources.

Code: 
Some IPs that resulted in 'sendto: operation not permitted':

65.203.23.136
66.111.4.55
74.125.95.104
64.233.169.104
66.249.81.104
67.195.160.76
72.30.2.43
72.14.204.103
72.14.204.147
72.14.204.104
72.14.204.99

I'm copying both my current firewall script and the old one below. What am I missing???
 
Original rule-set, with no traffic shaping or block lists:

Code: 
# Set variables.
#ext_if = "nfe0"
ext_if = "tun0"
#ext_if = "ng0"
lcl_if = "lo0"
lcl_ip = "127.0.0.1/32"
int_if = "rl0"
int_ip = "192.168.20.1/24"
int_sip = "192.168.20.1/32"
adm_if = "vr0"
adm_ip = "192.168.5.2/32"
adm_sip = "192.168.5.1/32"

pop3_ports = "{ 110, 995 }"
imap_ports = "{ 143, 993 }"
smtp_ports = "{ 25, 2225 }"
bind_ports = "{ 53 }"
webi_ports = "{ 80 }"
webo_ports = "{ 443 }"
admn_ports = "{ 23, 10101 }"
icmp_types = "echoreq"

# Set defaults.
set block-policy return
#set loginterface $ext_if
set skip on $lcl_if
scrub in on { $ext_if, $int_if }

# Allow internal interfaces to get to the internet.
nat on $ext_if from $int_ip to any -> ($ext_if)
nat on $ext_if from $adm_ip to any -> ($ext_if)
#nat-anchor "ftp-proxy/*"

# Allow local nets to FTP out.  Needed???
#rdr-anchor "ftp-proxy/*"
#rdr on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
#rdr on $adm_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021

# Block everything unless later explicitly allowed.
block in

# Keep state for established connections.
pass out keep state

# Allow local nets to FTP out.  Needed???
#anchor "ftp-proxy/*"

# Protect against IP spoofing on local network segments.
antispoof quick for { $lcl_if $int_if $adm_if }

# Block inbound traffic from IPs not valid for each interface.
block in quick on ! $int_if inet from $int_ip to any
block in quick on ! $adm_if inet from $adm_ip to any

# Nobody else is me - block attempts to make us think so.
block in quick on $int_if inet from $int_sip to any
block in quick on $adm_if inet from $adm_sip to any

# Assume administrator knows what he is doing.  Not necessarily true...
pass in quick on $adm_if from $adm_ip

pass in on $ext_if inet proto tcp from any to ($ext_if) port $pop3_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $imap_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $smtp_ports flags S/SA keep state
pass in on $ext_if inet proto { tcp, udp } from any to ($ext_if) port $bind_ports
pass in on $ext_if inet proto tcp from any to ($ext_if) port $webi_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $webo_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $pop3_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $imap_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $smtp_ports flags S/SA keep state
pass in on $int_if inet proto { tcp, udp } from any to any port $bind_ports
pass in on $int_if inet proto tcp from any to any port $webi_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $webo_ports flags S/SA keep state
pass in inet proto icmp all icmp-type $icmp_types keep state
 
Current rule-set, which has both traffic shaping and block tables. (One block table is from emergingthreats.org and the other a plain text file I created from IPs of people who tried to break in via brute force password guessing.)

Code: 
# Set variables.
#ext_if = "nfe0"
ext_if = "tun0"
#ext_if = "ng0"
lcl_if = "lo0"
lcl_ip = "127.0.0.1/32"
int_if = "rl0"
int_ip = "192.168.20.1/24"
int_sip = "192.168.20.1/32"
adm_if = "vr0"
adm_ip = "192.168.5.2/32"
adm_sip = "192.168.5.1/32"

pop3_ports = "{ 110, 995 }"
imap_ports = "{ 143, 993 }"
mail_ports = "{ 110, 995, 143, 993 }"
smtp_ports = "{ 25, 2225 }"
bind_ports = "{ 53 }"
webi_ports = "{ 80 }"
webo_ports = "{ 443 }"
admn_ports = "{ 23, 10101 }"
icmp_types = "echoreq"

# Set up tables.
table <emerging-threats> persist file "/usr/local/etc/IPBlocks/EmergingThreats"
table <other-blocked> persist file "/usr/local/etc/IPBlocks/Others"

# Set defaults.
set block-policy return
#set loginterface $ext_if
set skip on $lcl_if
scrub in on { $ext_if, $int_if }

# Activate alternate queuing
altq on tun0 cbq bandwidth 512Kb queue { standard_out, dns_out, http_out, tcpack_out, \
                                         popimap_out, smtp_out, admintraffic_out, wifitraffic_out }

# Set up queues
queue standard_out bandwidth 72Kb priority 0 cbq(default, borrow)
queue dns_out bandwidth 20Kb priority 6 cbq(borrow)
queue http_out bandwidth 40Kb priority 5 cbq(borrow)
queue tcpack_out bandwidth 40Kb priority 7 cbq(borrow)
queue popimap_out bandwidth 100Kb priority 4 cbq(borrow)
queue smtp_out bandwidth 100Kb priority 3 cbq(borrow)
queue admintraffic_out bandwidth 90Kb priority 2 cbq(borrow)
queue wifitraffic_out bandwidth 50Kb priority 1 cbq(borrow)

# Allow internal interfaces to get to the internet.
nat on $ext_if from $int_ip to any -> ($ext_if)
nat on $ext_if from $adm_ip to any -> ($ext_if)
#nat-anchor "ftp-proxy/*"

# Allow local nets to FTP out.  Needed???
#rdr-anchor "ftp-proxy/*"
#rdr on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
#rdr on $adm_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021

# Block everything unless later explicitly allowed.
block in

# Keep state for established connections.
pass out keep state

# Allow local nets to FTP out.  Needed???
#anchor "ftp-proxy/*"

# Protect against IP spoofing on local network segments.
antispoof quick for { $lcl_if $int_if $adm_if }

# Block inbound traffic from IPs not valid for each interface.
block in quick on ! $int_if inet from $int_ip to any
block in quick on ! $adm_if inet from $adm_ip to any

# Nobody else is me - block attempts to make us think so.
block in quick on $int_if inet from $int_sip to any
block in quick on $adm_if inet from $adm_sip to any

# Assume administrator knows what he is doing.  Not necessarily true...
pass in quick on $adm_if from $adm_ip

# Set up the rest of the rules
pass in on $ext_if inet proto tcp from any to ($ext_if) port $pop3_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $imap_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $smtp_ports flags S/SA keep state
pass in on $ext_if inet proto { tcp, udp } from any to ($ext_if) port $bind_ports
pass in on $ext_if inet proto tcp from any to ($ext_if) port $webi_ports flags S/SA keep state
pass in on $ext_if inet proto tcp from any to ($ext_if) port $webo_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $pop3_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $imap_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $smtp_ports flags S/SA keep state
pass in on $int_if inet proto { tcp, udp } from any to any port $bind_ports
pass in on $int_if inet proto tcp from any to any port $webi_ports flags S/SA keep state
pass in on $int_if inet proto tcp from any to any port $webo_ports flags S/SA keep state
pass in inet proto icmp all icmp-type $icmp_types keep state

# Block IPs classed as threats by EmergingThreats.org
block in log on $ext_if from <emerging-threats> to any
block out log on $ext_if from any to <emerging-threats>

# Block IPs that have tried to hack me
block in log on $ext_if from <other-blocked> to any

pass out on $ext_if inet proto { tcp udp } from ($ext_if) to any port $bind_ports keep state queue (dns_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $webi_ports keep state queue (http_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $mail_ports keep state queue (popimap_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $smtp_ports keep state queue (smtp_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($adm_if) to any keep state queue (admintraffic_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($int_if) to any keep state queue (wifitraffic_out, tcpack_out)
 
Code: 
pass out on $ext_if inet proto { tcp udp } from ($ext_if) to any port $bind_ports keep state queue (dns_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $webi_ports keep state queue (http_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $mail_ports keep state queue (popimap_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($ext_if) to any port $smtp_ports keep state queue (smtp_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($adm_if) to any keep state queue (admintraffic_out, tcpack_out)
pass out on $ext_if inet proto tcp from ($int_if) to any keep state queue (wifitraffic_out, tcpack_out)
jmm, is this correct from ($int_if) to any? Haven't used such syntax
 
You must log in or register to reply here.
Back
Top