[PF]: NAT & RDR rules for jailed proxy services

[Beeblebrox]

A wise man should know when to admit failure. Since half of my hair is in the trash, it's time I asked help.

Objective: run proxies in a jail and only one service per jail. Divert Internal Net and localhost traffic to the proxy jails. Allow jail traffic AND traffic for non-jailed services through the router NIC (ExtIf).

The setup:

• Three interfaces: $ExtIf (192.168.1.10/24), $IntIf (192.168.2.1/26), lo0 (127.0.0.1/8).
• One cloned interface $JailIf to be used as Jail NIC. No IP or netmask assigned, lo2 aliases: 192.168.2.97/32, 192.168.2.98/32, 192.168.2.99/32, 192.168.2.100/32 (four jails).
• Jails need to send and receive traffic to
  • each other
  • {lo0, $IntIF}
  • $ExtIF
• Each jail has in <jail>/etc/rc.conf:

  defaultrouter="ExtIF's IP"

• Traffic for jailed services originating from {lo0, $IntIF} should be redirected to the $JailIf. We should also make sure that no packets try to escape through $ExtIF without our consent.
• Relevant pf.conf entries:

# ExtIf, IntIf, JailIf defined, also each jail and gateway IP's
 jdns="192.168.2.97"	# Jail for unbound-DNS & dnscrypt-proxy using 443
 jprvx="192.168.2.99"	# Jail for Privoxy
 jhttp="192.168.2.100"	# Jail for squid-like http cache
 gate="192.168.1.10"

# Define jailed ports & service numbers
 JailTCP="{53,80,443,8118,9050}"

# NAT & RDR rules  ## NO SKIP on lo0 ##
# On $JailIf, NAT all traffic defined as to be hosted, and only that traffic
 nat on $JailIf proto {tcp,udp} from !($JailIf) to any port $JailTCP -> $JailIf

# Since (all packets) in the $JailTCP set have already been forwarded to $JailIf, NAT everything else arriving on the gateway normally.
 nat on $ExtIf from any to $ExtIf -> $gate

# I need all DNS lookup queries to got to the jail running unbound. NAT already does this I think, so not needed any longer? rdr on $JailIf proto {tcp,udp} from any port domain -> $jdns

## NOW re-direct by port, traffic exiting each jail to the gateway IP
# Unbound DNS on .97
 rdr on $JailIf proto {tcp,udp} from $jdns to any port {domain,443} -> $gate

# Privoxy on .99
 rdr on $JailIf proto {tcp} from $jprvx to any port {80,443} -> $gate

# HTTP Cache on .100 = Not needed?
#jhttp (cache) -> jprvx (privoxy) traffic will be defined from http-cache config file.
# rdr on $JaIf proto tcp from any to $JaIf port 8118 -> $jhttp

# Ntpd time server for the LAN
 rdr on $IntIf proto udp from $IntNet to $IntIf port ntp -> $IntIf

The problem:

1. NAT and redirect rules fail. It looks like traffic does not reach the external network. For example, the DNS jail sends traffic out, but it looks like it's not routed to the gate. tcpdump shows:
   

    192.168.2.97.57472 > 208.67.220.220.443: [udp sum ok] UDP, length 45

   Where <IP>:443 is the traffic generated by dnscrypt-proxy; and unrelated (?).
   

   ethertype IPv4 (0x0800), length 74: (tos 0x1c, ttl 64, id 61312, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->72d6)!)

2. The aim of the first NAT rule, is to capture on $ExtIf any rogue packets trying to escape without passing through the jail gauntlet. A solution would need to take that concern into account. Maybe a block filter for packets not originating from $JailIf would be simpler?

EDIT: I try to monitor traffic on lo2 or pflog0 using tcpdump -nettvv and this ruleset. Nothing shows up when I try to access the HTTP cache (192.168.2.100) although the cache responds on the browser (nothing to the outside, just internal page). I wonder if lo2 need to be given a network stack? I mean no IP, but a netmask?

 

[Beeblebrox]

Any idea why tcpdump is not showing any packet information? Better yet, how can I view a full trace of packets bouncing around on lo0, lo2 and $ExtIf for the ports in question (80, 8080, etc.)?