PF PF (firewalls, NAT, redirection) + hierarchical jails

toddg

toddg

I'm trying to build an app stack on FreeBSD using hierarchical vnet jails. It looks like this:

Code: 
HOST
+ JAILS
    + lb_jail
    + backup_jail
    + dev_jail
    + stage_jail
    + prod_jail

Within each [dev|stage|prod]_jail, I will have:
Code: 
    [dev|stage|prod]_jail
        + webserver
        + database

I am still building the the first part of this and haven't gotten to the hierarchical jails yet:

Code: 
HOST
+ JAILS
    + lb_jail
    + backup_jail

My jails are in 10.0.0.0/24. I have tried including this net range in my 'martians' list of unroutable addresses to protect my external interface. But this leads to the jails losing external internet access (pf.conf below). Is there a way do differentiate traffic on my internal 10.0.0.0/24 network ($jailnet) from outside unroutable world traffic on 10.0.0.0/8? I'd like to still block baddies on the outside, but allow my internal networks full connectivity.

Additional questions to show how little I understand about networking:

Q: I've read that lo0 is short for local 0, and is a virtual interface. I've also read that creating a bridge is 'cloning' the lo0 interface. Does that mean that all virtual interfaces can see each other? What about if they are on different subnets, e.g 10. and 192.? If I create bridge0 on 10.0.0.0/24 and bridge1 on 192.168.1.0/24, can these two subnets talk to each other? Should they be able to? Or say it was 10.0.0.0/24 and 10.0.1.0/24, what would we expect there? My expectation is that the examples above are different network segments and should not be able to talk to each other without something connecting them.

Note: I have read (and have on my desk) the following: mwlucas' jails, zfs, advanced zfs, networking for systems administrators (ok, i haven't read that much of this one), storage essentials, absolute freebsd, and absolute open bsd. A have read about half of kozierok's tcp/ip guide (that things huge).
 
I'll highlight the confusing bits here:

Code: 
# --------------------------------------------------------------
# PF Options (must come first)
# --------------------------------------------------------------
set skip on { lo0, bridge0 }            # Do not filter loopback/bridge traffic
scrub in                                # Normalize inbound packets

# --------------------------------------------------------------
# Interface definitions
# --------------------------------------------------------------
ext_if="em0"
jailnet = "10.0.0.0/24"
icmp_types  = "{ unreach, echoreq }"
icmp6_types = "{ unreach, echoreq, timex, paramprob }"
inbound_tcp_services = "{ ssh, https }"
outbound_tcp_services = "{ ssh, domain, http, https, rsync }"
outbound_udp_services = "{ domain, ntp }"
#inbound_udp_services = "{ domain }"


# --------------------------------------------------------------
# Ignore non-routable ip addresses
# --------------------------------------------------------------

>>>> If I add 10.0.0.0/8 to martians here, then my jails lose internet connectivity <<<<<<<<<

martians = "{ \
  0.0.0.0/8, \
  127.0.0.0/8, \
  169.254.0.0/16, \
  172.16.0.0/12, \
  192.0.2.0/24, \
  240.0.0.0/4 }"

# --------------------------------------------------------------
# NAT / Translation (must appear before filtering)
# --------------------------------------------------------------

>>>> So this gives my jails access to the internet <<<<<<

nat on $ext_if inet from $jailnet to any -> ($ext_if)

# --------------------------------------------------------------
# Tables (persisted across reloads)
# --------------------------------------------------------------
table <blocked> persist

# --------------------------------------------------------------
# Default policy (filtering section)
# --------------------------------------------------------------
block all

# --------------------------------------------------------------
# Quick block rules – drops packets that match before any other rule
# --------------------------------------------------------------
block quick from <blocked>

>>>> these are the parts that blocks my jails if the 10.0.0.0/8 network is added to the martians block <<<
block in quick on $ext_if from $martians to any
block out quick on $ext_if from any to $martians

# --------------------------------------------------------------
# SSH rate‑limiting (filtering)
# --------------------------------------------------------------
pass quick proto { tcp, udp } from $ssh_source_ips to any port ssh \
    flags S/SA keep state \
    (max-src-conn 15, max-src-conn-rate 5/3, overload <blocked> flush global)

# --------------------------------------------------------------
# Services you want to expose (filtering)
# --------------------------------------------------------------
pass out on $ext_if proto tcp from { self, $jailnet } to any port $outbound_tcp_services
pass in on $ext_if proto tcp from any to any port $inbound_tcp_services

# --------------------------------------------------------------
# Allow DNS (UDP)
# --------------------------------------------------------------
pass out on $ext_if proto udp to port $outbound_udp_services

# Allow ICMP (unreach, redirect, timex, echo‑request)
pass inet proto icmp icmp-type $icmp_types
pass inet6 proto icmp6 icmp6-type $icmp6_types

# Allow traceroute
pass out on $ext_if inet proto udp to port 33433:33626 # ipv4
pass out on $ext_if inet6 proto udp to port 33433:33626 # ipv6
 
My gut idea is telling me that I should add another internal network interface to the pf/firewall. This way internal jailnet traffic enters the firewall through a different interface and can be matched or tagged or something so that it can be allowed out on the external interface.

I thought that's what the NAT was doing:

Code: 
nat on $ext_if inet from $jailnet to any -&gt; ($ext_if)
 
You must log in or register to reply here.
Back
Top