PF woes: 3 interfaces and jails.

p5ycho

p5ycho

Hi guys, i have some trouble getting my new NAS/Router/AP to work smoothly.

My new NAS setup has the following interfaces:
re0: connected to ADSL bridge and the WWW, DHCP.
em0: Gbit LAN, IPv4 192.168.0.1-5/24
iwi0: Intel 2915 a/b/g wireless LAN
lo0: loopback, IPs: 127.0.0.1, 172.16.0.1/24, 172.16.0.10-19/24
172.16.0.0/24 range is meant for jails.

For now i want to keep iwi0 out of the picture, looks like I have some firmware loading troubles.
Also, ignore the IPv6 stuff.

What i want:
All nets should be able to reach the WWW, so some NAT is in order. This works.
Jails should be separated from the rest of the system as much as possible, and have access to the WWW
All DNS traffic should go through dnscache (192.168.0.3)(jails too), dnscache forwards to my ISP's DNS servers.

Observations:
NAT already works, jails and clients behind em0 can reach the WWW.
The jails can not reach the dnscache.
If anyone sees mistakes, easier ways to do things,etc, I'm willing to learn :)

Code: 
# pf.conf from insomnia.benzedrine.cx
#
# Default gateway doing NAT for my home network, IPv6 tunnel endpoint, using
# queues for extended ackpri (so I can work while being /., for instance).
#
# The external interface is re0,  with a single routable address (DHCP).
# The internal interface is em0,  address 192.168.0.1-5 in network 192.168.0.0/24.
# The wireless interface is iwi0, address 192.168.0.6   in network 192.168.0.0/24.
#
# The IPv6 tunnel is explained in more detail on
#   http://www.benzedrine.cx/gif.txt
#
# ACK prioritizing is explained on
#   http://www.benzedrine.cx/ackpri.html

# macros
#
ext_if		= "re0"
int_if		= "em0"
loc_if		= "lo0"
jailnet		= "172.16.0.0/24"
unfiltered	= "{ em0, lo0:0 }"
services_tcp	= "{ ssh, http, https }"
dnscache	= "192.168.0.3"
#ipv6_net	= "{ 2001:470:1f00:ffff::475, 2001:470:1f00:670:0:0:0:0/64 }"
#tunnel_peer	= "64.71.128.82"

# jailmacros
#
mysqljail		= "172.16.0.10"
httpjail		= "172.16.0.11"
squeezecenterjail	= "172.16.0.12"
rtorrentjail		= "172.16.0.13"
sabnzbpjail		= "172.16.0.14"
#mysqlusers		= "{ 172.16.0.11 }"

# options
#
set loginterface $ext_if
set limit states 10000
set limit frags 500

# tables
#
table <scanners> persist
table <ext_unroutable> const { 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12\
			192.168.0.0/16, 255.255.255.255/32 }


# normalization
#
scrub in  on $ext_if all           fragment reassemble
scrub out on $ext_if all random-id fragment reassemble

# queues
#
# - ssh has maximum priority, so the host is always managable.
# - dns has high priority.
# - outgoing tcp connections have priority over incoming ones.
# - outgoing http has priority over other outgoing tcp
#   (the mailing list generates bursts of outgoing smtp)
#
# effect: even while the web server is under heavy load and the
# mailing list is delivering mails, I can still use www/icb/irc.
#
altq on $ext_if priq bandwidth 1Mb queue { q_max, q_hig, q_def, q_low }
queue q_max priority 7
queue q_hig priority 5
queue q_def priority 3
queue q_low priority 1 priq(default)

# translations
#

no nat on $unfiltered from any to any
no rdr on $unfiltered from any to any

# nat jails and private network to single routable address
#
nat on $ext_if inet from $jailnet to any -> $ext_if:0
nat on $ext_if inet from $int_if:network to any -> $ext_if:0

# redirection
#
rdr on $ext_if proto tcp from any to $ext_if port http \
	-> $httpjail port http
rdr on $ext_if proto tcp from any to $ext_if port https \
	-> $httpjail port https

# =============================================================================
# filter rules (default block/pass)
# =============================================================================

# pass on unfiltered interfaces
#
pass quick on $unfiltered

# silently drop TCP non-SYN packets, the remaining ruleset only deals with
# TCP SYNs, which always create state when passed. the ruleset basically
# deals with 'connections', not packets, beyond this point.
#
block return-rst quick proto tcp all flags /S
block return-rst quick proto tcp all flags A/A

# block and log everything by default
#
block             log
block return-rst  log inet proto tcp
block return-icmp log inet proto udp

# =============================================================================
# loopback interface (all local/jail-to-jail IPv4 traffic)
# =============================================================================

#pass quick on $int_if inet proto tcp from $mysqlusers to $mysqljail port 3306 \
        keep state queue (q_low, q_def)

#!!!!!
#pass quick on $jailnet inet proto tcp from any to any  \
        keep state queue (q_low, q_def)
#!!!!!

#pass quick on $loc_if:0 inet proto tcp from lo0 to $jailnet port ssh \
        keep state queue (q_low, q_def)
#pass quick on $int_if inet from $jailnet to $int_if \
        keep state queue (q_low, q_def)
#pass out on $ext_if inet proto udp from $dns to any port domain \
	keep state queue (q_hig)


# =============================================================================
# external interface (all external IPv4 traffic)
# =============================================================================

# silently drop broadcasts (Net noise)
#
block in quick on $ext_if inet from any to { 255.255.255.255, $ext_if:broadcast }

# block unwanted peers
#
block in quick on $ext_if inet from <scanners> to any

# block some known-bad ports without logging
#
block return-rst  in quick on $ext_if proto tcp from any to any \
	port { 111, 445, 1080, 6000, 6667 }
block return-icmp in quick on $ext_if proto udp from any to any \
	port { 137, 138, 139, 1434 }

# block and log incoming packets from reserved address space and invalid
# addresses, they are either spoofed or misconfigured, we can't reply to
# them anyway (hence, no return-rst).
#
block in log quick on $ext_if inet from <ext_unroutable> to any

# block and log outgoing packets that don't have my address as source, they are
# either spoofed or something is misconfigured (NAT disabled, for instance),
# we want to be nice and not send out garbage.
#
block out log quick on $ext_if inet from !$ext_if to any

# ICMP
#
pass out on $ext_if inet proto icmp from $ext_if to any \
	icmp-type 8 code 0 keep state queue (q_max)

pass in  on $ext_if inet proto icmp from any to $ext_if \
	icmp-type 8 code 0 keep state (max 32) queue (q_low)

# UDP
#
pass out on $ext_if inet proto udp from any to any \
	keep state queue (q_def)
#all dns goes through dnscache.
#pass out on $ext_if inet proto udp from $dnscache to any port domain \
	keep state queue (q_hig)

#pass in  on $ext_if inet proto udp from any to $ext_if port $rtorrent \
	keep state (max 1024, source-track rule, max-src-nodes 512, max-src-states 5) \
	queue q_low

#Quake3 server
#pass in proto udp from any to any port 27960 keep state
# no public dns configured
#pass in  on $ext_if inet proto udp from any to $ext_if \
	port domain keep state (max 512) queue (q_hig)

# TCP
#
pass out on $ext_if inet proto tcp from $ext_if to any \
	flags S/SA modulate state queue (q_def, q_max)
pass out on $ext_if inet proto tcp from $ext_if to any port http \
	flags S/SA keep state queue (q_hig, q_max)

#'from !<scanners>' is useless?, already dropped?
pass in  quick on $ext_if inet proto tcp from !<scanners> to $ext_if port ssh \
	flags S/SA keep state (max-src-conn 5, max-src-conn-rate 4/120, \
	overload <scanners> flush) queue (q_def, q_max)
pass in  on $ext_if inet proto tcp from any to $ext_if port $services_tcp \
	flags S/SA keep state (max 1024, tcp.first 10, tcp.opening 10) \
	queue (q_low, q_max)
pass in  on $ext_if inet proto tcp from any to $httpjail port {http, https} \
	keep state queue (q_low, q_def)

# other protocols (IPv6 tunnel)
#
#pass out on $ext_if inet proto ipv6 from $ext_if to $tunnel_peer \
#	keep state (other.multiple 86400) queue (q_def)
#pass in  on $ext_if inet proto ipv6 from $tunnel_peer to $ext_if \
#	keep state (other.multiple 86400) queue (q_def)

# =============================================================================
# tunnel interface (all external IPv6 traffic)
# =============================================================================

# ICMP
#
#pass out on gif0 inet6 proto ipv6-icmp from $ipv6_net to any \
#	icmp6-type echoreq keep state queue (q_max) allow-opts

#pass in  on gif0 inet6 proto ipv6-icmp from any to $ipv6_net \
#	icmp6-type echoreq keep state (max 32) queue (q_low)

# UDP
#
#pass out on gif0 inet6 proto udp from $ipv6_net to any \
#	keep state queue (q_def)
#pass out on gif0 inet6 proto udp from $ipv6_net to any port domain \
#	keep state queue (q_hig)

#pass in  on gif0 inet6 proto udp from any to $ipv6_net \
#	port domain keep state (max 512) queue (q_hig)

# TCP
#
#pass out on gif0 inet6 proto tcp from $ipv6_net to any flags S/SA \
#	keep state queue (q_def, q_max)

#pass in  on gif0 inet6 proto tcp from any to $ipv6_net port $services_tcp \
#	flags S/SA keep state (max 1024, tcp.first 20, tcp.opening 20) \
#	queue (q_low, q_max)
 
If your $ext_if gets it's IP address from DHCP use ($ext_if). That will use the interface's IP address even if it changes.
 
It's DHCP, but its locked to the mac address. So that makes it pretty static :)

Any clue as to why my jails can't connect to 192.168.0.3?
 
There are no rules defining traffic from the jails to $int_if (where 192.168.0.3 is).
 
Code: 
#pass quick on $int_if inet from $jailnet to $int_if \
        keep state queue (q_low, q_def)
It was commented. Doesn't matter if I comment or not though :)

I've been looking at it for too long, if someone could give me some pointers as to what I'm doing right and what definitely needs improving/can be done more efficiently, please do so :)
 
I'm trying to get a ping from $jailnet to $dnscache like this:
Code: 
pass out quick on $loc_if inet proto icmp from $jailnet to $dnscache \
        icmp-type 8 code 0 keep state (max 32) queue (q_low)

pfctl -s state | grep -i icmp states:
Code: 
all icmp 172.16.0.12:20518 -> 192.168.0.3       0:0
then it occurred to me, all traffic outside $jailnet probably ends up being NATed through $ext_if?
Should I use a redirect rule to fix this?
 
p5ycho said:
then it occurred to me, all traffic outside $jailnet probably ends up being NATed through $ext_if?
Click to expand...
Ah, yes. This one:
Code: 
nat on $ext_if inet from $jailnet to any -> $ext_if:0

Should I use a redirect rule to fix this?
Click to expand...
You could try changing the NAT to something like:
Code: 
nat on $ext_if inet from $jailnet to !$dnscache -> $ext_if:0
Not sure if it'll work though, never used anything like that :e
 
I'm being stupid here. The solution i'm looking for is layer 2 bridging, and i'll go and take a look in the handbook before i try anything else.

I'll be back when i have more insight in this matter :)
 
It's pretty annoying that editing messages is enabled. Sorry for the many posts.

Looks like i did the right thing above, but i forgot to allow dnscache lookups from the 172.16.0 range. :OOO
 
You must log in or register to reply here.
Back
Top