PF NAT isn't re-writing my jail network addresses as expected, causing ping to fail b/c no response

toddg

toddg

How do I configure NAT so that my jails can access the internet? My jail is nating using it's gateway address, but isn't translating that gateway to a routable address. I'm guessing it has something to do with the fact that my GUEST.em0 interface is inside the pubbridge bridge. The jail is sending traffic out it's default gateway, 10.0.0.1, but I must not be natting from that gateway to em0?

Here's my setup:
Code: 
LAPTOP (FreeBSD 15.0) -> GUEST (VirtualBox running FreeBSD 15.1 OS) -> Jail (FreeBSD 15.1)

I have tcpdump capturing packets at LAPTOP.re0 and GUEST.em0. icmp tests show:

FROM -> TO : icmp packets
laptop -> 8.8.8.8 : yes
guest -> laptop -> 8.8.8.8 : yes
jail -> guest -> laptop -> 8.8.8.8 : yes // but the packets from the jail show an origination as the jail's gateway 10.0.0.1... that's an un-routable address...shouldn't NAT have re-written it?
8.8.8.8 -> laptop : yes
8.8.8.8 -> laptop -> guest : yes
8.8.8.8 -> laptop -> guest -> jail : no

Here's some more info on my setup:

LAPTOP
FreeBSD system76 15.0-RELEASE-p12 FreeBSD 15.0-RELEASE-p12 releng/15.0-n281099-7b527b9b97ba GENERIC amd64

LAPTOP: rc.conf
Bash: 
hostname="foobar"
ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
vboxnet_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"
moused_port=/dev/psm0
moused_enable=YES
ntpdate_enable="YES"
ntpdate_flags="-u pool.ntp.org"
nfs_client_enable=YES
rpc_statd_enable=YES
rpc_lockd_enable=YES
autofs_enable=YES
devd_enable=YES
devfs_system_ruleset="system"
dbus_enable=YES
kld_list="fusefs nvidia-drm"
ifconfig_wlan0="WPA DHCP"

LAPTOP: ifconfig
Bash: 
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether d4:93:90:3c:84:69
    inet 192.168.0.230 netmask 0xffffff00 broadcast 192.168.0.255
    inet6 fe80::d693:90ff:fe3c:8469%re0 prefixlen 64 scopeid 0x1
    inet6 2601:602:8100:d9a0:d693:90ff:fe3c:8469 prefixlen 64 autoconf pltime 229139 vltime 229139
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

GUEST (jail host) : rc.conf
Bash: 
hostname="server_a"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
moused_nondefault_enable="NO"
dumpdev="AUTO"
zfs_enable="YES"
pf_enable="YES"
pf_fallback_rules_enable="YES"
pf_fallback_rules="block drop log all
pass quick on em0"
pflog_enable="YES"
syslogd_flags="-ss"
jail_enable="YES"
jail_parallel_start="NO"
gateway_enable="YES"
ipv6_gateway_enable="NO"
ifconfig_em0="up"
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0_name="pubbridge"
ifconfig_bridge1_name="prvbridge"
ifconfig_pubbridge="addm em0 SYNCDHCP up descr 'public jails bridge, connects to external interface'"
ifconfig_pubbridge_alias0="inet 10.0.0.1 netmask 0xFFFFFF00"
ifconfig_prvbridge="descr 'private bridge, no connection to external interface' up"

GUEST (jail host) : pf.conf
Bash: 
# --------------------------------------------------------------
# Macros
# --------------------------------------------------------------
# The external interface has been put into a bridge, so reference
# the bridge as the external interface going forward.
ext_if          = "pubbridge"

# JAIL_HOST pf.conf
jail_net        = "{ 10.0.0.1/24 }"

# Jail groupings

glb_jails = "{ 10.0.0.2 }"

# ICMP types
icmp_types                    = "{ unreach, echoreq }"
icmp6_types                   = "{ unreach, echoreq, timex, paramprob routeradv, neighbradv, neighbrsol }"

# Ingress Groupings

prometheus_tcp_ingress = "{ 192.168.0.0/16 }"
nodeexporter_tcp_ingress = "{ 192.168.0.0/16 }"
ssh_tcp_ingress = "{ 192.168.0.0/16 }"
http_tcp_ingress = "{ 192.168.0.0/16 }"
https_tcp_ingress = "{ 192.168.0.0/16 }"

# The book of pf, 4th edition page 93, defines martians as non-routable
# ip address (RFC-1918) traffic on an external interface. Of course we
# want to block that bullshit. But we also don't want to block our internal
# 10.0.0.0/28 traffic, either.
# TODO: figure out how to block external 10.0.0.0/28 traffic but allow
# TODO: internal 10.0.0.0./28 and internal 192.168.0.0/24 traffic
martian_nets = "{
172.16.0.0/12,0.0.0.0/8,127.0.0.0/8
}"


# --------------------------------------------------------------
# Tables
# --------------------------------------------------------------
# Dynamic lists of addresses
# martians table
table <martians> { $martian_nets }
# bruteforce blackhole for bad ips
table <bruteforce> persist
# table for the entire interal jail network
table <jails_v4> { $jail_net }
# jail tables

table <glb_jails> { $glb_jails }
# ingress tables

table <prometheus_tcp_ingress> { $prometheus_tcp_ingress }
table <nodeexporter_tcp_ingress> { $nodeexporter_tcp_ingress }
table <ssh_tcp_ingress> { $ssh_tcp_ingress }
table <http_tcp_ingress> { $http_tcp_ingress }
table <https_tcp_ingress> { $https_tcp_ingress }


# --------------------------------------------------------------
# Options
# --------------------------------------------------------------
# Global behavior settings
set skip on lo0
set block-policy drop
set loginterface $ext_if
logopt = "log"
logall = "log"


# --------------------------------------------------------------
# Ethernet
# --------------------------------------------------------------
#
# --------------------------------------------------------------
# Normalization
# --------------------------------------------------------------
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --------------------------------------------------------------
# Queueing
# --------------------------------------------------------------

# --------------------------------------------------------------
# Translations (NAT)
# --------------------------------------------------------------
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# --------------------------------------------------------------
# Filtering
# --------------------------------------------------------------
# Port redirections

rdr pass on $ext_if inet proto tcp from <http_tcp_ingress> to ($ext_if) port 80 -> <glb_jails> port 8080
rdr pass on $ext_if inet proto tcp from <https_tcp_ingress> to ($ext_if) port 443 -> <glb_jails> port 8443

# Default deny everything
block log all



# Allow port redirections from external clients to the jail

pass in quick on $ext_if inet proto tcp from <http_tcp_ingress> to <glb_jails> port 8080 keep state label port_fwd
pass in quick on $ext_if inet proto tcp from <https_tcp_ingress> to <glb_jails> port 8443 keep state label port_fwd

# Ingress

pass in quick on $ext_if proto tcp from <prometheus_tcp_ingress> to any port { 9090 3000 }
pass in quick on $ext_if proto tcp from <nodeexporter_tcp_ingress> to any port 9100
pass in quick on $ext_if proto tcp from <ssh_tcp_ingress> to any port 22 flags S/SA keep state (max-src-conn 5, max-src-conn-rate 3/30 overload <bruteforce> flush global) label 'allow_ssh_from_trusted'
pass in quick on $ext_if proto tcp from <http_tcp_ingress> to any port 80
pass in quick on $ext_if proto tcp from <https_tcp_ingress> to any port 443

# Jail to Jail Traffic


# Block known bad actors immediately
block quick from <bruteforce>
block in quick on $ext_if from <martians> to any
block out quick on $ext_if from any to <martians>

# Allow all established connections out
pass out on $ext_if proto { tcp, udp } from ($ext_if) modulate state

# Anti-spoofing
antispoof quick for { $ext_if }

# Block and log all other SSH attempts (for monitoring)
block in log quick on $ext_if proto tcp from any to any port 22 label "ssh_blocked"

# Useful ICMP
pass log inet proto icmp icmp-type $icmp_types
pass inet6 proto icmp6 icmp6-type $icmp6_types


# Jail Egress

pass in quick on $ext_if proto tcp from <glb_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <glb_jails> to any port { 443 } keep state # packages


# Enable DNS Queries
# enable outbound dns queries from the external interface
pass out quick on $ext_if proto { tcp udp } from ($ext_if)  to any port 53 keep state

# TODO is this required? If not then close it
# enable inbound dns queries to the external interface from the jails
pass in  quick on $ext_if proto { tcp udp } from <jails_v4> to any port 53 keep state

# TEMPORARY : GIVE ALL JAILS FULL EGRESS
pass in quick on $ext_if from <jails_v4> to ! <jails_v4> keep state

GUEST (jail host) : ifconfig
Bash: 
toddg@server_a ~ $ ifconfig
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS>
    ether 08:00:27:7f:a5:81
    inet6 fe80::a00:27ff:fe7f:a581%em0 prefixlen 64 scopeid 0x1
    inet6 2601:602:8100:d9a0:a00:27ff:fe7f:a581 prefixlen 64 autoconf pltime 229139 vltime 229139
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pubbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: public jails bridge, connects to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: ep_stgjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 18 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_prdjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 14 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_glb0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 10 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_devjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 6 priority 128 path cost 2000 vlan protocol 802.1q
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
prvbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: private bridge, no connection to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:57:49
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
    options=0
    groups: pflog
ep_devjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:devjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:78:27
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_glb0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:glb0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:a9:c6
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:bd:bc
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_stgjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:stgjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:61:02
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

GUEST (jail host) : jls
Bash: 
toddg@server_a ~ $ jls
   JID  IP Address      Hostname                      Path
     1                  devjh0                        /jails/containers/devjh0
     2                  glb0                          /jails/containers/glb0
     3                  prdjh0                        /jails/containers/prdjh0
     4                  stgjh0                        /jails/containers/stgjh0

JAIL ifconfig
Code: 
root@glb0 / $ ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0 metric 0 mtu 33152
    options=0
    groups: pflog
ep_glb0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:37:76
    inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail ARP
Code: 
root@glb0 / $ arp -na
? (10.0.0.2) at 58:9c:fc:10:37:76 on ep_glb0b permanent [ethernet]
? (10.0.0.1) at 58:9c:fc:10:7e:44 on ep_glb0b expires in 524 seconds [ethernet]

JAIL Netstat
Code: 
root@glb0 / $ netstat -arn
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            10.0.0.1           UGS        ep_glb0b
10.0.0.0/24        link#11            U          ep_glb0b
10.0.0.2           link#12            UHS             lo0
127.0.0.1          link#12            UH              lo0

Here's what tcpdumps are capturing while i'm running pings from both the jail, the guest, and the laptop:

GUEST em0 tcpdump
Code: 
toddg@server_a ~ $ sudo tcpdump -n -e -ttt -i em0 | grep -i icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on em0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 00:00:00.019205 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 20, length 64
 00:00:00.000201 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 18124, seq 587, length 64
 00:00:00.011344 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 20, length 64
 00:00:00.010748 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 21, length 64
 00:00:00.000112 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 18124, seq 588, length 64
 00:00:00.011859 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 21, length 64
 00:00:00.007728 a4:01:de:2f:38:95 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 198: fe80::a601:deff:fe2f:3895 > ff02::1: ICMP6, router advertisement, length 144
 00:00:00.000103 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 22, length 64
 00:00:00.000038 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 18124, seq 589, length 64
 00:00:00.010987 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 22, length 64
^C362 packets captured
400 packets received by filter
0 packets dropped by kernel

LAPTOP re0 tcpdump
Code: 
$ sudo tcpdump -n -e -ttt -i re0 | grep -i icmp
...
 00:00:00.195978 d4:93:90:3c:84:69 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.230 > 8.8.8.8: ICMP echo request, id 14369, seq 144, length 64
 00:00:00.010988 a4:01:de:2f:38:95 > d4:93:90:3c:84:69, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.230: ICMP echo reply, id 14369, seq 144, length 64
 00:00:00.183349 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 11383, seq 3, length 64
 00:00:00.610894 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 268, length 64
 00:00:00.011264 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 268, length 64
 00:00:00.234684 d4:93:90:3c:84:69 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.230 > 8.8.8.8: ICMP echo request, id 14369, seq 145, length 64
 00:00:00.012756 a4:01:de:2f:38:95 > d4:93:90:3c:84:69, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.230: ICMP echo reply, id 14369, seq 145, length 64
 00:00:00.125514 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 11383, seq 4, length 64
 00:00:00.000400 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 269, length 64
 00:00:00.011320 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 269, length 64
 00:00:00.190911 d4:93:90:3c:84:69 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.230 > 8.8.8.8: ICMP echo request, id 14369, seq 146, length 64
 00:00:00.012459 a4:01:de:2f:38:95 > d4:93:90:3c:84:69, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.230: ICMP echo reply, id 14369, seq 146, length 64
 00:00:00.054228 a4:01:de:2f:38:95 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 198: fe80::a601:deff:fe2f:3895 > ff02::1: ICMP6, router advertisement, length 144
 00:00:00.123029 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 11383, seq 5, length 64
 00:00:00.119316 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 37147, seq 270, length 64
 00:00:00.013492 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 37147, seq 270, length 64
 00:00:00.128123 d4:93:90:3c:84:69 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.230 > 8.8.8.8: ICMP echo request, id 14369, seq 147, length 64
^C786 packets captured
794 packets received by filter
0 packets dropped by kernel

So you can see that only the jail is not getting icmp responses, and that's certainly b/c it is sending the icmp requests as 10.0.0.1, an un-routable address. So my question is, what's wrong with my NAT stanza in my pf.conf?
 
Ok, so I thought fixed the icmp ping part of this with the following:

1. Fix pf.conf to use the jail IPs, not gateway in jail_net
Code: 
jail_net        = "10.0.0.0/24"              # Match entire subnet (not the jail gateway of 10.0.0.1/24)
2. Nat on the interface, not the bridge
Code: 
nat on em0 inet from 10.0.0.0/24 to any -> (em0) # nat on the external interface em0, not the bridge pubbridge

Still testing, but that doesn't seem to have fixed it.
 
I have simplified my setup to one jail, in a vm, trying to ping 8.8.8.8.

I get the following behavior consistently:
* The first time I ping from a JAIL, it fails.
* The second time I ping from the JAIL, it succeeds.
* During the first ping, NAT is replacing the JAIL ip address with the gateway address. This is an unroutable address.
* During the second ping, NAT is replacing the JAIL ip address with the HOST ip address (and this succeeds).

My questions:
* Um, help? Any ideas what I'm doing wrong?

JAIL (10.0.0.2)
Bash: 
root@glblb0 / $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 0 packets received, 100.0% packet loss
root@glblb0 / $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=13.071 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=14.119 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=12.051 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=10.384 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=114 time=13.161 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=114 time=14.930 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.384/12.953/14.930/1.457 ms

So the first time I try to ping, 100% packet loss. The second time, immediately after pressing CTRL-C, 0% packet loss. The tcpdump shows why:

SERVER_A (virtual box vm)(192.168.0.40)
Bash: 
$ sudo tcpdump -n -e -ttt -i em0 | grep -i icmp | egrep -v advertisement
listening on em0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
 00:00:00.010099 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 10.0.0.1 > 8.8.8.8: ICMP echo request, id 61845, seq 5, length 64
 # ... more requests, no replies
 00:00:00.004863 58:9c:fc:10:7e:44 > a4:01:de:2f:38:95, ethertype IPv4 (0x0800), length 98: 192.168.0.40 > 8.8.8.8: ICMP echo request, id 10498, seq 0, length 64
 00:00:00.000069 a4:01:de:2f:38:95 > 58:9c:fc:10:7e:44, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.0.40: ICMP echo reply, id 10498, seq 0, length 64
 # ... more requests, more replies

This looks to me like something weird with NAT. I'll attach relevant files, but here's the relevant parts in pf.conf
Bash: 
ext_if          = "pubbridge"
jail_net        = "{ 10.0.0.1/24 }"
table <jails_v4> { $jail_net }
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

And here's the relevant parts of server_a's ifconfig
Bash: 
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS>
    ether 08:00:27:7f:a5:81
    inet6 fe80::a00:27ff:fe7f:a581%em0 prefixlen 64 scopeid 0x1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pubbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: public jails bridge, connects to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: ep_glblb0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 6 priority 128 path cost 2000 vlan protocol 802.1q
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
ep_glblb0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:glblb0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:78:27
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

----
Files pasted in below
----

/etc/pf.conf
Code: 
# --------------------------------------------------------------
# Macros
# --------------------------------------------------------------
# The external interface has been put into a bridge, so reference
# the bridge as the external interface going forward.
# the raw_if references the actual network interface on the host
raw_if          = "em0"

# The external interface that raw_if points to has been put into a bridge, "pubbridge".
# Reference the pubbridge as the external interface going forward.
ext_if          = "pubbridge"

# JAIL_HOST pf.conf
jail_net        = "{ 10.0.0.1/24 }"

glblb_jails = "{ 10.0.0.2 }"


icmp_types                    = "{ unreach, echoreq }"
icmp6_types                   = "{ unreach, echoreq, timex, paramprob }"


prometheus_tcp_ingress = "{ 192.168.0.0/16 }"
nodeexporter_tcp_ingress = "{ 192.168.0.0/16 }"
ssh_tcp_ingress = "{ 192.168.0.0/16 }"
http_tcp_ingress = "{ 192.0.0.0/8 }"
https_tcp_ingress = "{ 192.0.0.0/8 }"

# The book of pf, 4th edition page 93, defines martians as non-routable
# ip address (RFC-1918) traffic on an external interface. Of course we
# want to block that bullshit. But we also don't want to block our internal
# 10.0.0.0/28 traffic, either.
# TODO: figure out how to block external 10.0.0.0/28 traffic but allow
# TODO: internal 10.0.0.0./28 and internal 192.168.0.0/24 traffic
martian_nets = "{
172.16.0.0/12,0.0.0.0/8,127.0.0.0/8
}"


# --------------------------------------------------------------
# Tables
# --------------------------------------------------------------
# Dynamic lists of addresses
# martians table
table <martians> { $martian_nets }
# bruteforce blackhole for bad ips
table <bruteforce> persist
# table for the entire interal jail network
table <jails_v4> { $jail_net }
# jail tables

table <glblb_jails> { $glblb_jails }
# ingress tables

table <prometheus_tcp_ingress> { $prometheus_tcp_ingress }
table <nodeexporter_tcp_ingress> { $nodeexporter_tcp_ingress }
table <ssh_tcp_ingress> { $ssh_tcp_ingress }
table <http_tcp_ingress> { $http_tcp_ingress }
table <https_tcp_ingress> { $https_tcp_ingress }


# --------------------------------------------------------------
# Options
# --------------------------------------------------------------
# Global behavior settings
set skip on lo0
set block-policy drop
set loginterface $ext_if

# --------------------------------------------------------------
# Ethernet
# --------------------------------------------------------------
#
# --------------------------------------------------------------
# Normalization
# --------------------------------------------------------------
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --------------------------------------------------------------
# Queueing
# --------------------------------------------------------------

# --------------------------------------------------------------
# Translations (NAT)
# --------------------------------------------------------------
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

# --------------------------------------------------------------
# Filtering
# --------------------------------------------------------------
# Port redirections

rdr pass on $ext_if inet proto tcp from <http_tcp_ingress> to ($ext_if) port 80 -> <glblb_jails> port 8080
rdr pass on $ext_if inet proto tcp from <https_tcp_ingress> to ($ext_if) port 443 -> <glblb_jails> port 8443

# Default deny everything
block log all

# Block known bad actors immediately
block quick from <bruteforce>
block in quick on $ext_if from <martians> to any
block out quick on $ext_if from any to <martians>

# Allow port redirections from external clients to the jail

pass in quick on $ext_if inet proto tcp from <http_tcp_ingress> to <glblb_jails> port 8080 keep state label port_fwd
pass in quick on $ext_if inet proto tcp from <https_tcp_ingress> to <glblb_jails> port 8443 keep state label port_fwd

# Ingress

pass in quick on $ext_if proto tcp from <prometheus_tcp_ingress> to any port { 9090 3000 }
pass in quick on $ext_if proto tcp from <nodeexporter_tcp_ingress> to any port 9100
pass in quick on $ext_if proto tcp from <ssh_tcp_ingress> to any port 22 flags S/SA keep state (max-src-conn 5, max-src-conn-rate 5/30 overload <bruteforce> flush global) label 'allow_ssh_from_trusted'
pass in quick on $ext_if proto tcp from <http_tcp_ingress> to any port 80
pass in quick on $ext_if proto tcp from <https_tcp_ingress> to any port 443

# Jail to Jail Traffic


# Allow all established connections out
pass out on $ext_if proto { tcp, udp, icmp } from ($ext_if) modulate state

# Anti-spoofing
antispoof quick for { $ext_if }

# Block and log all other SSH attempts (for monitoring)
block in log quick on $ext_if proto tcp from any to any port 22 label "ssh_blocked"

# Useful ICMP
pass inet proto icmp icmp-type $icmp_types
pass inet6 proto icmp6 icmp6-type $icmp6_types


# Jail Egress

pass in quick on $ext_if proto tcp from <glblb_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <glblb_jails> to any port { 443 } keep state # packages


# Enable DNS Queries
# enable outbound queries from the external interface
pass out quick on $ext_if proto { tcp udp } from ($ext_if)  to any port domain
# enable inbound quires to the external interface from the jails
pass in  quick on $ext_if proto { tcp udp } from <jails_v4> to any port domain

# TEMPORARY : GIVE ALL JAILS FULL EGRESS
pass in quick on $ext_if from <jails_v4> to ! <jails_v4> keep state

/etc/rc.conf
Code: 
hostname="server_a"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
pf_enable="YES"
pf_fallback_rules_enable="YES"

# multi-rule
pf_fallback_rules="block drop log all
pass quick on em0"

pflog_enable="YES"
node_exporter_enable="NO"

syslogd_flags="-ss"
defaultrouter="192.168.0.1"

# jails
jail_enable="YES"
jail_parallel_start="NO"
gateway_enable="YES"
ipv6_gateway_enable="NO"

ifconfig_em0="up"

# create a bridges for the host jails
cloned_interfaces="bridge0 bridge1"

ifconfig_bridge0_name="pubbridge"
ifconfig_bridge1_name="prvbridge"

# public bridge
ifconfig_pubbridge="addm em0 SYNCDHCP up descr 'public jails bridge, connects to external interface'"

ifconfig_pubbridge_alias0="inet 10.0.0.1 netmask 0xFFFFFF00"

# private bridges

ifconfig_prvbridge="descr 'private bridge, no connection to external interface' up"

full ifconfig output
Bash: 
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS>
    ether 08:00:27:7f:a5:81
    inet6 fe80::a00:27ff:fe7f:a581%em0 prefixlen 64 scopeid 0x1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pubbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: public jails bridge, connects to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: ep_glblb0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 6 priority 128 path cost 2000 vlan protocol 802.1q
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
prvbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: private bridge, no connection to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:57:49
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
ep_glblb0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:glblb0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:78:27
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

netstat
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ netstat -arn -4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.0.1        UGS       pubbridge
10.0.0.0/24        link#3             U         pubbridge
10.0.0.1           link#2             UHS             lo0
127.0.0.1          link#2             UH              lo0
192.168.0.0/24     link#3             U         pubbridge
192.168.0.40       link#2             UHS             lo0

arp
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ arp -na
? (192.168.0.40) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (192.168.0.1) at a4:01:de:2f:38:95 on pubbridge expires in 1160 seconds [bridge]
? (10.0.0.1) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (192.168.0.230) at d4:93:90:3c:84:69 on pubbridge expires in 560 seconds [bridge]

/etc/sysctl.conf
Code: 
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.init_shutdown_timeout="900"
# enable forwarding
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
# this would be nice, but it's experimental, so keep it disabled
net.link.bridge.inherit_mac=0
# enable pf on bridge
net.link.bridge.pfil_bridge=1
# standard vnet jails stuff
net.link.bridge.pfil_member=0
net.link.bridge.pfil_onlyip=0
# standard zfs config setting
vfs.zfs.vdev.min_auto_ashift=12
 
Fixed

/etc/pf.conf
Git: 
- nat on $ext_if inet from <jails_v4> to any -> ($ext_if)
+ nat on $ext_if inet from <jails_v4> to any -> ($ext_if:0)

Explanation
(from LLM, take it with salt)

Code: 
This isn't routing or filtering — it's your NAT rule behaving exactly as written, and the clue is that ($ext_if) doesn't mean "the address of the interface." It means "every address on the interface."
What's Happening

pubbridge has two addresses: 192.168.0.40 (primary, DHCP) and 10.0.0.1 (alias, the jail gateway). So this rule:
nat on $ext_if inet from <jails_v4> to any -> ($ext_if)

expands into a source address pool containing both addresses, and pf's default pool type for NAT is round-robin. Each time a new state is created, pf picks the next address in the pool:

    First ping: new state → pool starts at 10.0.0.1 → NAT rewrites 10.0.0.2 to 10.0.0.1. Your upstream router (192.168.0.1) only knows about 192.168.0.0/24, so packets with source 10.0.0.1 go out the door but the replies never come back (upstream has no route to 10.0.0.1). 100% loss.
    Ctrl-C: the ICMP state closes.
    Second ping: new state → round-robin rotates to the next pool address, 192.168.0.40 → NAT works, replies flow.

That's why it's so perfectly deterministic: odd attempts get one pool address, even attempts the other. (With only two addresses it's first-fails-second-works; with more addresses you'd see alternating success/failure continue.)
The Fix

Pin the NAT to the interface's primary address with the :0 suffix:
nat on $ext_if inet from <jails_v4> to any -> ($ext_if:0)

($ext_if:0) means "the first (primary, non-alias) address of the interface" — which is your DHCP-assigned 192.168.0.40 — and unlike hardcoding the IP, it still tracks DHCP changes dynamically.

LLM SUGGESTED LINKS/SOURCES
* https://man.freebsd.org/cgi/man.cgi?query=pf.conf
* https://docs.freebsd.org/en/books/handbook/firewalls/
* https://www.openbsd.org/faq/pf/
* https://www.bsdly.net/~peter/pf.html
 
wait! You load pf rules inside your jails? I use (per jail) anchors on my host; am I doing something wrong (are my anchors not doing anything per se)?
 
JohnK So far, I'm only running pf on my host, not in my jails. And for that, I don't use anchors (yet) b/c I want to see everything in one file, pf.conf. Since I generate pf.conf anyway, it's more convenient for me that way... but I think anchors would be smarter. Eventually I'll probably use them.

Once I get these jails playing nicely with the host, I will be running pf in these jails... but more on that once I get there.

Right now I'm having an issue with NAT breaking when I have multiple aliases on my bridge... I'll write up a post on that next in this thread.
 
I’m in the process of checking my setup (breaking one of my servers now). I’ll post something outlining my method (may not be 100pct correct but the overview should suffice later for a reference). I automate my anchors.

I also use bridge so I don’t fully understand all the differences between that and nat but…
 
JohnK Here's what I have going atm:

HOST jails
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ jls
   JID  IP Address      Hostname                      Path
    24                  devjh0                        /jails/containers/devjh0
    25                  glblb0                        /jails/containers/glblb0
    26                  prdjh0                        /jails/containers/prdjh0
    27                  stgjh0                        /jails/containers/stgjh0
    28                  prdjh1                        /jails/containers/prdjh1
    29                  prdjh2                        /jails/containers/prdjh2
    30                  prdjh3                        /jails/containers/prdjh3
    31                  prdjh4                        /jails/containers/prdjh4

HOST ifconfig
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ ifconfig
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS>
    ether 08:00:27:7f:a5:81
    inet6 fe80::a00:27ff:fe7f:a581%em0 prefixlen 64 scopeid 0x1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pubbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: public jails bridge, connects to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:7e:44
    inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
    inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
    inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    member: ep_prdjh4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 34 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_prdjh3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 30 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_prdjh2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 26 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_prdjh1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 22 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_stgjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 18 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_prdjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 14 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_glblb0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 10 priority 128 path cost 2000 vlan protocol 802.1q
    member: ep_devjh0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 6 priority 128 path cost 2000 vlan protocol 802.1q
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            port 1 priority 128 path cost 20000 vlan protocol 802.1q
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
prvbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: private bridge, no connection to external interface
    options=10<VLAN_HWTAGGING>
    ether 58:9c:fc:10:57:49
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    bridge flags=0<>
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
ep_devjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:devjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:78:27
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_glblb0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:glblb0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:a9:c6
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:bd:bc
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_stgjh0a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:stgjh0
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:61:02
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh1a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh1
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:5a:12
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh2a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh2
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:1c:52
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh3a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh3
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:d6:e2
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ep_prdjh4a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: jail:vnet:prdjh4
    options=20001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,RXCSUM_IPV6>
    ether 58:9c:fc:10:63:b6
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

HOST arp
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ arp -na
? (192.168.0.40) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (10.0.0.2) at 58:9c:fc:10:37:76 on pubbridge expires in 182 seconds [bridge]
? (10.0.3.1) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (10.0.1.2) at 58:9c:fc:10:78:1c on pubbridge expires in 312 seconds [bridge]
? (192.168.0.1) at a4:01:de:2f:38:95 on pubbridge expires in 1165 seconds [bridge]
? (10.0.2.1) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (10.0.1.1) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (10.0.0.1) at 58:9c:fc:10:7e:44 on pubbridge permanent [bridge]
? (192.168.0.230) at d4:93:90:3c:84:69 on pubbridge expires in 685 seconds [bridge]
? (10.0.3.6) at 58:9c:fc:10:61:10 on pubbridge expires in 1116 seconds [bridge]

HOST netstat
Code: 
toddg@server_a ~/repos/biz/freebsd/cloud $ netstat -arn
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.0.1        UGS       pubbridge
10.0.0.0/24        link#3             U         pubbridge
10.0.0.1           link#2             UHS             lo0
10.0.1.0/24        link#3             U         pubbridge
10.0.1.1           link#2             UHS             lo0
10.0.2.0/24        link#3             U         pubbridge
10.0.2.1           link#2             UHS             lo0
10.0.3.0/24        link#3             U         pubbridge
10.0.3.1           link#2             UHS             lo0
127.0.0.1          link#2             UH              lo0
192.168.0.0/24     link#3             U         pubbridge
192.168.0.40       link#2             UHS             lo0

HOST /etc/rc.conf
Code: 
hostname="server_a"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
pf_enable="YES"
pf_fallback_rules_enable="YES"
# multi-rule
pf_fallback_rules="block drop log all
pass quick on em0"
pflog_enable="YES"
node_exporter_enable="NO"
syslogd_flags="-ss"
# jails
jail_enable="YES"
jail_parallel_start="NO"
gateway_enable="YES"
ipv6_gateway_enable="NO"
ifconfig_em0="up"
# create a bridges for the host jails
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0_name="pubbridge"
ifconfig_bridge1_name="prvbridge"
# public bridge
ifconfig_pubbridge="addm em0 SYNCDHCP up descr 'public jails bridge, connects to external interface'"
ifconfig_pubbridge_alias0="inet 10.0.0.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias1="inet 10.0.1.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias2="inet 10.0.2.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias3="inet 10.0.3.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias4="inet 10.0.3.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias5="inet 10.0.3.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias6="inet 10.0.3.1 netmask 0xFFFFFF00"
ifconfig_pubbridge_alias7="inet 10.0.3.1 netmask 0xFFFFFF00"
# private bridges
ifconfig_prvbridge="descr 'private bridge, no connection to external interface' up"

HOST /etc/pf.conf
Code: 
# --------------------------------------------------------------
# Macros
# --------------------------------------------------------------
# The external interface has been put into a bridge, so reference
# the bridge as the external interface going forward.
# the raw_if references the actual network interface on the host
raw_if          = "em0"

# The external interface that raw_if points to has been put into a bridge, "pubbridge".
# Reference the pubbridge as the external interface going forward.
ext_if          = "pubbridge"

# JAIL_HOST pf.conf
jail_net        = "{ 10.0.0.0/24,10.0.1.0/24,10.0.2.0/24,10.0.3.0/24,10.0.3.0/24,10.0.3.0/24,10.0.3.0/24,10.0.3.0/24 }"

devjh_jails = "{ 10.0.1.2 }"
glblb_jails = "{ 10.0.0.2 }"
prdjh_jails = "{ 10.0.3.6,10.0.3.5,10.0.3.4,10.0.3.3,10.0.3.2 }"
stgjh_jails = "{ 10.0.2.2 }"

# ICMP types
icmp_types                    = "{ unreach, echoreq }"
icmp6_types                   = "{ unreach, echoreq, timex, paramprob }"

# Ingress Groupings

prometheus_tcp_ingress = "{ 192.168.0.0/16 }"
nodeexporter_tcp_ingress = "{ 192.168.0.0/16 }"
ssh_tcp_ingress = "{ 192.168.0.0/16 }"
http_tcp_ingress = "{ 192.168.0.0/16 }"
https_tcp_ingress = "{ 192.168.0.0/16 }"

# The book of pf, 4th edition page 93, defines martians as non-routable
# ip address (RFC-1918) traffic on an external interface. Of course we
# want to block that bullshit. But we also don't want to block our internal
# 10.0.0.0/28 traffic, either.
# TODO: figure out how to block external 10.0.0.0/28 traffic but allow
# TODO: internal 10.0.0.0./28 and internal 192.168.0.0/24 traffic
martian_nets = "{
172.16.0.0/12,0.0.0.0/8,127.0.0.0/8
}"


# --------------------------------------------------------------
# Tables
# --------------------------------------------------------------
# Dynamic lists of addresses
# martians table
table <martians> { $martian_nets }
# bruteforce blackhole for bad ips
table <bruteforce> persist
# table for the entire interal jail network
table <jails_v4> { $jail_net }
# jail tables

table <devjh_jails> { $devjh_jails }
table <glblb_jails> { $glblb_jails }
table <prdjh_jails> { $prdjh_jails }
table <stgjh_jails> { $stgjh_jails }
# ingress tables

table <prometheus_tcp_ingress> { $prometheus_tcp_ingress }
table <nodeexporter_tcp_ingress> { $nodeexporter_tcp_ingress }
table <ssh_tcp_ingress> { $ssh_tcp_ingress }
table <http_tcp_ingress> { $http_tcp_ingress }
table <https_tcp_ingress> { $https_tcp_ingress }


# --------------------------------------------------------------
# Options
# --------------------------------------------------------------
# Global behavior settings
set skip on lo0
set block-policy drop
set loginterface $ext_if
logopt = "log"
logall = "log"


# --------------------------------------------------------------
# Ethernet
# --------------------------------------------------------------
#
# --------------------------------------------------------------
# Normalization
# --------------------------------------------------------------
scrub in all fragment reassemble
scrub out all random-id max-mss 1500

# --------------------------------------------------------------
# Queuing
# --------------------------------------------------------------

# --------------------------------------------------------------
# Translations (NAT)
# --------------------------------------------------------------
nat on $ext_if inet from <jails_v4> to any -> ($ext_if:0)

# --------------------------------------------------------------
# Filtering
# --------------------------------------------------------------
# Port redirections

rdr pass on $ext_if inet proto tcp from <http_tcp_ingress> to ($ext_if) port 80 -> <glblb_jails> port 8080
rdr pass on $ext_if inet proto tcp from <https_tcp_ingress> to ($ext_if) port 443 -> <glblb_jails> port 8443

# Default deny everything
block log all

# Block known bad actors immediately
block quick from <bruteforce>
block in quick on $ext_if from <martians> to any
block out quick on $ext_if from any to <martians>

# Allow port redirections from external clients to the jail

pass in quick on $ext_if inet proto tcp from <http_tcp_ingress> to <glblb_jails> port 8080 keep state label port_fwd
pass in quick on $ext_if inet proto tcp from <https_tcp_ingress> to <glblb_jails> port 8443 keep state label port_fwd

# Ingress

pass in quick on $ext_if proto tcp from <prometheus_tcp_ingress> to any port { 9090 3000 }
pass in quick on $ext_if proto tcp from <nodeexporter_tcp_ingress> to any port 9100
pass in quick on $ext_if proto tcp from <ssh_tcp_ingress> to any port 22 flags S/SA keep state (max-src-conn 5, max-src-conn-rate 5/30 overload <bruteforce> flush global) label 'allow_ssh_from_trusted'
pass in quick on $ext_if proto tcp from <http_tcp_ingress> to any port 80
pass in quick on $ext_if proto tcp from <https_tcp_ingress> to any port 443

# Jail to Jail Traffic

pass in quick on $ext_if proto tcp from <glblb_jails> to <devjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <stgjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <prdjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <prdjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <prdjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <prdjh_jails> port { 8080 8443 } keep state
pass in quick on $ext_if proto tcp from <glblb_jails> to <prdjh_jails> port { 8080 8443 } keep state

# Allow all established connections out
pass out on $ext_if proto { tcp, udp } from ($ext_if) modulate state

# Anti-spoofing
antispoof quick for { $ext_if }

# Block and log all other SSH attempts (for monitoring)
block in log quick on $ext_if proto tcp from any to any port 22 label "ssh_blocked"

# Useful ICMP
pass log inet proto icmp icmp-type $icmp_types
pass inet6 proto icmp6 icmp6-type $icmp6_types


# Jail Egress

pass in quick on $ext_if proto tcp from <devjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <devjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <glblb_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <glblb_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <prdjh_jails> to any port { 443 } keep state # packages

pass in quick on $ext_if proto tcp from <stgjh_jails> to any port { 80 } keep state # packages

pass in quick on $ext_if proto tcp from <stgjh_jails> to any port { 443 } keep state # packages


# Enable DNS Queries
# enable outbound dns queries from the external interface
pass out quick on $ext_if proto { tcp udp } from ($ext_if)  to any port domain keep state

# TODO is this required? If not then close it
# enable inbound dns queries to the external interface from the jails
pass in  quick on $ext_if proto { tcp udp } from <jails_v4> to any port domain keep state

# TEMPORARY : GIVE ALL JAILS FULL EGRESS
pass in quick on $ext_if from <jails_v4> to ! <jails_v4> keep state

Now the Jails have full access to the internet. I'll want to lock that down later to just what's needed, but I'll roll with that for now.
The jails also have full access to each other's subnets, which is something that I want to lock down such that:

* the load balancer jails in 10.0.0.0 can talk to each of the network segments: 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24, and 10.0.3.0/24.
* the other jails (dev|stg|prd) can only access their own network segments, e.g. jails in 10.0.1.0/24 cannot access jails in 10.0.2.0/24.

I thought this would have done that:
Code: 
pass in quick on $ext_if from <jails_v4> to ! <jails_v4> keep state

So that's tomorrows project, isolating the jails from each other.
 
I googled NAT vs Bridge and from what I understand (I lost the link already) is that NAT does not allow the container to talk to anything else besides the host (one way). If you want each container to be able to talk to each other I think you need bridge (but please know that I only googled this just now so, I'm really not educated on that at all).


However, I wanted to post my method (although, I'm not entirely sure this is a viable method at the moment so I need to find a way to test if this method works for sure or not).

My pf.conf (snips):

Code: 
# Macros
# ----------------------------------------------------------------------
#       User-defined variables may be defined and used later.
ext_if = "em0"                     # main interface
bridge_if = "bridge0"              # bridge interface for jails

...

# Tables
# ----------------------------------------------------------------------
#      Tables provide a mechanism for increasing the performance and
#      flexibility of rules with large numbers of source or
#      destination addresses.
# NOTE:
# pfctl provides the ability to expire table entries. For example,
# this command will remove <bruteforce> table entries which have not
# been referenced for 86400 seconds:
#      pfctl -t bruteforce -T expire 86400
table <jails> persist

# Anchors
# ----------------------------------------------------------------------
#      An anchor is a container that can hold rules, address tables,
#      and other anchors.
anchor "jails/*"       # load individual jail anchors with: pfctl -a jails/<name> -f /etc/pf/jails/anchors/<name>.conf

 # NOTE:
 # The concept for the jail table and anchors (per jail) is to create a rule and list out
 #  # EXAMPLE (/etc/pf/jails/anchor/pf.emby.conf):
 #  # pass in on $ext_if proto tcp to 192.168.0.11 port 8096 flags S/SA keep state
 # then ADD the jail's IP when the jail starts: pfctl -t jails -T add 192.168.0.11
 # with the jail.conf file's exec.prestart command.
 # REMOVE the jail's IP the jail stops: pfctl -t jails -T delete 192.168.0.111
 # with the jail.conf exec.poststop command.
 #
 # Example:
 # Add jail's ip in jail.conf: exec.prestart += "pfctl -t jails -T add 192.168.0.11 || true"
 # Remove jail's IP on stop: exec.poststop += "pfctl -t jails -T delete 192.168.0.11 || true"

# Packet Filtering
# ----------------------------------------------------------------------
block in all                       # Block all incomming by default.
...

Where a jail anchor looks like:
Code: 
## @file /etc/pf/jails/anchors/pf.git.conf
## @brief Anchor rules for Git jail (jail IP: 192.168.0.2)
## @date April 7, 2026
##
## Load: pfctl -a jails/git -f /etc/pf/jails/anchors/pf.git.conf
## Unload: pfctl -a jails/git -F rules
## Test: pfctl -n -a jails/git -f /etc/pf/jails/anchors/pf.git.conf
## View Rules: pfctl -a jails/git -s rules

bridge_if = "bridge0"              # bridge interface for jails
lan_net = "192.168.0.0/24"         # LAN network covering jails
host_ip = "192.168.0.2"            # server public/host IP (on bridge0_alias0)
ssh_port = "22"                    # host SSH port (non-default)

# Allow SSH connections to git.local from local lan only.
pass in quick on $bridge_if proto tcp from $lan_net to $host_ip port $ssh_port \
   flags S/SA keep state \
   (max-src-conn 15, max-src-conn-rate 5/3)

And my /etc/jail.conf.d/git.conf looks like:
Code: 
git {
  $id = 2;

  exec.prestart += "if [ -f /opt/conf/sys/pf/anchors/pf.git.conf ]; then cp /opt/conf/sys/pf/anchors/pf.git.conf /etc/pf/jails/anchors/pf.git.conf; fi";
  exec.prestart += "pfctl -t jails -T add 192.168.0.2 || true";

  exec.prestop += "if [ -f /etc/pf/jails/anchors/pf.git.conf ]; then rm /etc/pf/jails/anchors/pf.git.conf; fi";
  exec.poststop += "pfctl -t jails -T delete 192.168.0.2 || true";

  # MOUNTS
  ...
}
 
JohnK Hey, that's nice and clean. I esp. like how your comments help the reader with various tools to use in the future.

Bash: 
## @file /etc/pf/jails/anchors/pf.git.conf
## @brief Anchor rules for Git jail (jail IP: 192.168.0.2)
## @date April 7, 2026
##
## Load: pfctl -a jails/git -f /etc/pf/jails/anchors/pf.git.conf
## Unload: pfctl -a jails/git -F rules
## Test: pfctl -n -a jails/git -f /etc/pf/jails/anchors/pf.git.conf
## View Rules: pfctl -a jails/git -s rules

6 months from now, those load/unload/test/view commands will be **really** handy.

Regarding your anchor, I wonder if you could move everything here except the jail's ip address to the parent pf.conf?

Bash: 
bridge_if = "bridge0"              # bridge interface for jails
lan_net = "192.168.0.0/24"         # LAN network covering jails
host_ip = "192.168.0.2"            # server public/host IP (on bridge0_alias0)
ssh_port = "22"                    # host SSH port (non-default)

# Allow SSH connections to git.local from local lan only.
pass in quick on $bridge_if proto tcp from $lan_net to $host_ip port $ssh_port \
   flags S/SA keep state \
   (max-src-conn 15, max-src-conn-rate 5/3)

I haven't tested, but I think the anchors are evaluated within the context of the parent pf.conf (and parent anchors), so that could become:
Bash: 
host_ip = "192.168.0.2"            # server public/host IP (on bridge0_alias0)

# Allow SSH connections to git.local from local lan only.
pass in quick on $bridge_if proto tcp from $lan_net to $host_ip port $ssh_port \
   flags S/SA keep state \
   (max-src-conn 15, max-src-conn-rate 5/3)
 
toddg said:
esp. like how your comments help the reader with various tools to use in the future.
Click to expand...
HA! you mean my "dummy comments"? Yes, I make tons of those for myself. Use them; you have the winning formula (they are required to become a member of "team awesome") now but the only added rules for them is (which you currently cannot grok):
1. if the file is long (long as in you'd have to scroll up if you did a cat(1) on the file you need to place them at the bottom (as well))
2. The DOxy style isn't required but can make grep(1) easier.
I add them to most files (even the one file C programs like:
C: 
@build cc -std=c99 -02 -Wall -lm -o ...
@usage ./myprog /path/to/socket.sock
@usage (produce a report): printf 'REPORT\n' | nc -U /path/to/socket.sock

toddg said:
the anchors are evaluated within the context of the parent pf.conf
Click to expand...
I think the same thing. I'm not on BSD at the moment so I'll run some tests later. ...this example (my git server) is a bit overkill because that ssh rule is part of my main pf.conf so having it in the git anchor is redundant but I wanted something a bit easier for my post here.

I need clean because in my jail making (wrapper) script I have several optional key/values in my jail setup configs for "prestart, prestop, etc..." for adding to the /etc/jail.conf.d/git.conf file. However, the script only reads the first entry so, if you need to preform several steps on the host system my script currently will not perform them all (another reason why I'm rewriting my script in C). But technically speaking, the anchors can exist in the anchors directory and just not be loaded/added/deleted in the jail.conf.d/git.conf file (I could technically remove two rules from my git.conf file--I only preform them so that I can make modifications to the anchor, restart the jail to load in the most current anchor rules).

REF:
github.com

jcreate/doc/jcreate.7 at 0b4cedd6dd780ee5d54bfc30780845886361318d · JohnKaul/jcreate

Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities - JohnKaul/jcreate
github.com github.com
 
My original NAT issue has been resolved. For anyone getting tripped up by this in the future, here's a tar of all the files I generated for my system. It's actually a tar.gz, so rename it.
 

Attachments

You must log in or register to reply here.
Back
Top