Solved PF not working on FreeBSD 10.1 with 2 NICs

micski

micski

PF not working on FreeBSD 10.1 with 2 NICs

If you have a dedicated FreeBSD 10.1 server at home, with 2 NICs, connecting WAN and LAN, what would a simple PF configuration look like?

It might sound like a RTFM question, but, after weeks of PF manual, FreeBSD handbook, websites and reboots, I only get a complete network black-out, when FreeBSD boots with IPFW disabled and PF enabled. Not even the following basic test configuration works. IPFW and NAT works (but with dead slow upload, when NAT is used).

rc.conf
Code: 
...
pf_enable="YES"
pf_rules="/root/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""
...

/root/pf.conf
Code: 
lan = "bce0"
wan = "bge0"
table <blacklist> file "/root/pf-blacklist.txt"
scrub in on $wan all fragment reassemble
nat on $wan from $lan:network to any -> ($wan)
block all
block in log quick on $wan from <blacklist> to any
pass in on $lan
pass out on $wan

Reason for change to PF is the slow speed problem (5 kb/s) with IPFW and some Xen, Intel and other NICs. Confirmed by RootBSD.
 
You can use the following as a template. Basic idea is to block and log all traffic that is not allowed (yet). By running tcpdump on the pflog0 interface you can see the blocked packets.
We filter on the internal interface and tag or label the traffic we want to pass.
On the external interface we only pass this tagged traffic.

Because I don't remember the for me as OpenBSD pfuser the "old" NAT syntax on FreeBSD somebody else will have to help out on that ;)

Code: 
external = re0
internal = lo0

# ---- default policy
block log all

# --- NAT rule

# --- EXTERNAL INTERFACE
# -- outgoing
pass out quick on $external tagged OUT_OK


# --- INTERNAL INTERFACE
# --- incoming
pass in quick on $internal inet proto udp to port domain  tag OUT_OK
pass in quick on $internal inet proto tcp to port www     tag OUT_OK

PS You will have to adapt the definitions of the external and internal macros. I just used those to verify the syntax.
 
Here is another trivial config to try. What have you done to troubleshoot so far? Try adding log statements to all your rules and run tcpdump -n -e -ttt -i pflog0 as shown in pflog(4) to see it in real time. Look at what rules are actually getting hit with pfctl -vs rules. Is forwarding/gateway enabled? Post up your results if you can't figure it out.
Code: 
lan_ifs = "{" wlan0 bridge0 tun0 tun1 tun2 em0 em1 "}"
wan_phy = "em3"
nat on $wan_phy inet from !($wan_phy) to any -> ($wan_phy:0)
block log all
pass out keep state
pass in on $lan_ifs
 
Hello,

I ran FreeBSD 10.1-RELEASE (amd64) on a Shuttle DS437 at home which has an RTL8111G network chipset. The box has two ethernet port, and I managed to make a router without having the issues you have. You can find my full firewall ruleset here as an example. I assume you have the line gateway_enable="YES" in /etc/rc.conf. Did you try a single line ruleset such as pass all keep state just to check the result? (with an additional NAT rule obviously, I didn't mention it).

Regards,
Guillaume
 
Thanks a lot for looking into this.

My home server is a HP ProLiant ML.

# uname -s -r -p
Code: 
FreeBSD 10.1-RELEASE amd64

# dmesg
Code: 
...
brgphy0: <BCM5708C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Ethernet address: 00:23:7d:5c:5a:b0
...
brgphy1: <BCM5722 1000BASE-T media interface> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:25:b3:b8:54:22
...

# grep gateway /etc/rc.conf
Code: 
gateway_enable="YES"

# grep pf /etc/rc.conf
Code: 
pf_enable="YES"
pf_rules="/root/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""

Should the following single line ruleset work? I am not sure, if PF will translate only packets, going out - and not in. Keep in mind, that the system routes fine with IPFW. IPFW is, naturally, disabled while testing PF. The IPFW configuration has 2 translation rules.

Code: 
# Macros
lan = "bce0"
wan = "bge0"

# Translation
nat on $wan from $lan:network to any -> ($wan)

# Packet Filtering
pass all keep state
 
The "keep state" isn't strictly necessary, it's on by default. It will allow responses to a packet. So if certain packets are allowed out their responses are automatically allowed back in and vise verse. That's the nature of a stateful firewall.
 
I have now rebooted and tested the single rule configuration above. The test was not a succes; The entire network is still blacked-out with PF.

The workstations can not ping other host IP addresses than themselves. The server can not ping other host IP addresses either. "Permission denied". The server can not ping hostnames either. "Hostname lookup failure".

When I disable PF, re-enable IPFW and reboot, everything works again?
 
The workstation (192.168.1.100) can ping itself 192.168.1.100. The workstation can not ping the gateway/server (192.168.1.1). The server can not ping the workstation nor Google.

I did not test, wheither one workstation can ping another workstation.

I find it interesting, that "permission denied" was given at the ping test. It indicates, that PF is blocking.

The LAN interface is connected to a switch in which workstations and other LAN devices are attached.

The WAN interface is connected to a bridge to the ISP DSLAM.

Maybe we can invite Dutch into this thread? I read in another thread, that he also has this particular setup at home and is using FreeBSD/PF.
 
Is that workstation able to ping other workstations in the same network?

After making changes to pf.conf did you re-load it?
 
I wrote the new pf.conf, disabled IPFW, disabled NAT, enabled PF in rc.conf and rebooted the system. Is that the correct procedure? Or should I compile pf.conf into PF before reboot?
 
This server has to work after a reboot. Therefore, PF must route packets and work, when the server is booted.

I will test your load/reload procedure at next test.
 
micski said:
This server has to work after a reboot. Therefore, PF must route packets and work, when the server is booted.
Click to expand...
Sure, I get that. But it's a bit of a chore to keep rebooting the server just to load a new ruleset. Especially when you're testing things. Just use the reload function and reboot one last time when you think you've got everything worked out.

Also make use of pfctl -nf /etc/pf.conf. This will load the rules, check its syntax but does not actually apply them. This will prevent you from trying a non-working ruleset.
 
In case you are running in securelevel above -1, you may have trouble loading the pf kernel module. If that is the case, can you try to add to /boot/loader.conf the lines:
Code: 
# Load PF kernel modules
pf_load="YES"
pflog_load="YES"
Also, the command mentioned by SirDice should return an error if the kernel modules are not loaded. Just a thought.

Regards,
Guillaume
 
pfctl -nf /etc/pf.conf did not return any errors nor warnings.

Configuring pf and pflog to load from boot/loader.conf made the difference. Also, I noticed, that dummynet was loaded. I do not know, if this could have caused the boot proces to activate ipfw. I removed it.

I also noted, that using pf and this basic ruleset configuration solved my slow upload issue; The server now has its fast upload speed restored.

Thanks for the hints. Now, I just need to write a proper ruleset configuration for pf.
 
Hi, micski!
I have 2 NICs, one is the external IP and another one is for internal. This setup worked in 10.0. I upgraded to 10.1, now it is not working for the port redirecting. It seems that port 80 is blocked. I can telnet <MY IP> 8280, but not telnet <MY IP> 80

Then I disabled pf(4) and changed Apache to listen to 80. Then I can telnet 127.0.0.1 80, I cannot telnet <MY IP> 80 from external.

Can you help me to figure out what is the problem?
Code: 
ext_if="re1"
int_if="re0"
set skip on lo
scrub in
nat on $ext_if inet from !($ext_if) -> ($ext_if:0)
rdr pass on $ext_if proto tcp to port 80 -> 127.0.0.1 port 8280
antispoof quick for { lo $int_if }
root@www:/etc # pfctl -vnf /etc/pf.conf
Code: 
ext_if = "re1"
int_if = "re0"
set skip on { lo }
scrub in all fragment reassemble
nat on re1 inet from ! (re1) to any -> (re1:0)
rdr pass on re1 inet proto tcp from any to any port = http -> 127.0.0.1 port 8280
block drop in quick on ! lo inet6 from ::1 to any
block drop in quick on ! lo inet from 127.0.0.0/8 to any
block drop in quick inet from 127.0.0.1 to any
block drop in quick on ! re0 inet from 192.168.0.0/24 to any
block drop in quick inet from 192.168.0.1 to any
block drop in quick inet6 from ::1 to any
block drop in quick on lo0 inet6 from fe80::1 to any
# uname -a
Code: 
FreeBSD www.pccom.ca 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0 r279896M: Wed Mar 11 23:43:25 EDT 2015
peng@www.pccom.ca:/usr/obj/usr/src/sys/GENERIC  amd64
 
Last edited by a moderator:
frankpeng, does pfctl -vsn show that rule making states?
Example:
Code: 
rdr pass on tun0 inet proto tcp from any to ! <firewall> port = ftp -> 127.0.0.1 port 8021
  [ Evaluations: 433672    Packets: 34        Bytes: 2372        [FILE]States: 1[/FILE]     ]
  [ Inserted: uid 0 pid 41934 [FILE]State Creations: 1[/FILE]     ]

Consider a split rule so it is easier to see the counter hits. The rdr hits will be shown with pfctl -vsn and normal rule hits shown with pfctl -vsr. You shouldn't have too much trouble tracking the issue down with the verbose output.
/etc/pf.conf
Code: 
rdr on $ext_if inet proto tcp from any to any port 80 -> 127.0.0.1 port 8280
pass in on $ext_if inet proto tcp to any port 8280
 
May I ask why you're running www services on a non standard port and why you're redirecting it to localhost? It just adds overhead...
//Danne
 
Hi, junovitch!
I was kicking my 10.1 from outside with telnet pccom.ca 80
It displayed like this:
Code: 
Trying 209.141.130.62 ...
telnet: connet to address 209.141.130.62: operation time out
telnet: Unable to connect to remote host
While doing this:
In my 10.1, it looked like this:
root@www:/etc # pfctl -vsn
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
nat on re1 inet from ! (re1) to any -> (re1:0)
  [ Evaluations: 766       Packets: 48        Bytes: 9678        States: 3     ]
  [ Inserted: uid 0 pid 2050 State Creations: 4     ]
rdr pass on re1 inet proto tcp from any to any port = http -> 127.0.0.1 port 8280
  [ Evaluations: 1161      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
root@www:/etc # pfctl -vsr
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
scrub in all fragment reassemble
  [ Evaluations: 4047      Packets: 2413      Bytes: 1133785     States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick on ! lo inet6 from ::1 to any
  [ Evaluations: 1955      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick on ! lo inet from 127.0.0.0/8 to any
  [ Evaluations: 1178      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick inet from 127.0.0.1 to any
  [ Evaluations: 1178      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick on ! re0 inet from 192.168.0.0/24 to any
  [ Evaluations: 1178      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick inet from 192.168.0.1 to any
  [ Evaluations: 1178      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick inet6 from ::1 to any
  [ Evaluations: 1178      Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
block drop in quick on lo0 inet6 from fe80::1 to any
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 2050 State Creations: 0     ]
root@www:/etc #
Then from outside:
wget pccom.ca not working wget pccom.ca:8280 works.

I called my ISP. They are saying they were not blocking port 80 incoming.
 
diizzy said:
May I ask why you're running www services on a non standard port and why you're redirecting it to localhost? It just adds overhead...
//Danne
Click to expand...
To run www on a non standard port is for security.
To redirect to port 80 is to allow use to use pccom.ca instead of pccom.ca:8280 to access the web site.
 
frankpeng said:
To run www on a non standard port is for security.
To redirect to port 80 is to allow use to use pccom.ca instead of pccom.ca:8280 to access the web site.
Click to expand...

What you're doing adds no security at all. You are exposing the service to internet on TCP port 80 of the public IP address you're advertising the service to be on. Whatever tricks you're playing with port redirections doesn't change that fact and doesn't make the work for a potential attacker any harder in case there are vulnerabilities that can be exploited in the service.
 
People (me too) believe to run any service at port greater than 1024 is more secure than ports less than 1024. Some service even refuse to run at port less than 1024. Some people scared to run services at ports less than port 1024. I do not know why.
 
You must log in or register to reply here.
Back
Top