PF PF + VNET iocage jails

Recently I changed my jails to being VNET jails, and I thought I checked to make sure it was working okay with my firewall, but apparently not. It seems my firewall rules are doing much of nothing. I have a somewhat complex setup with my networking. I have my iocage jails using a bridged interface, in addition to that I have bhyve jails using that same bridge. One of my rules is to block an <externaljails> table from communicating with an <internaljails> table. That isn't working. Additionally, I also tried blocking traffic from one jail to my network (not shown in the contents below), but that seemed to do nothing, no matter what I tried. Did I royally mess up my firewall rules? Seems this is the #1 issue I always return to with FreeBSD. Definitely my own problem, but I've read and researched and can't seem to get this right. Here are my rules

Code:
ext_if="em0"
localnet="192.168.10.0/24"

nextcloud = "{ 192.168.10.42 }"
githost = "192.168.10.50"
plex = "{ 192.168.10.44 }"

## Set and drop these IP ranges on public interface ##
table <martians> { 127.0.0.0/8 172.16.0.0/12 10.0.0.0/8 192.168.0.0/16 \
              169.254.0.0/16 192.0.2.0/24 0.0.0.0/8 240.0.0.0/4 255.255.255.255/32 \
              !192.168.10.0/24 !127.0.0.1 }

webports = "{ http, https }"
tcp_services = " { bootps, bootpc, domain, ntp }"
udp_services = "{ bootps, bootpc, domain, ntp }"
udp6_services="{ 53, 123, 1194, 546, domain }" # 546 == dhcpv6-client
plex_ports_tcp = "{ 32400, 3005, 8324, 32469 }"
plex_ports_udp = "{ 1900, 5353, 32410, 32412, 32413, 32414 }"

icmp_types = "{echoreq, unreach}"
icmp6_types="{ 128, 133, 134, 135, 136, 137 }"

tcp_state="flags S/SA keep state"
udp_state="keep state"

table <bruteforce> persist
table <goodhosts> { 192.168.10.0/24 }
table <externaljails> { $nextcloud, $githost }
table <internaljails> { $plex }
table <sshjails> { 192.168.10.50 }


# Don't send rejections. Just drop.
set block-policy return

# Exempt the loopback interface to prevent services utilizing the
# local loop from being blocked accidentally.
set skip on lo0

# all incoming traffic on external interface is normalized and fragmented
# packets are reassembled.
scrub in on $ext_if all fragment reassemble

# set a default deny policy.
block in log all

# Drop all Non-Routable Addresses
block drop in quick on $ext_if from <martians> to any
block drop out quick on $ext_if from any to <martians>

# Enable antispoofing on the external interface
antispoof quick for $ext_if

block in on $ext_if from urpf-failed to any

# Block internal and external jails from communicating
block drop quick from <externaljails> to <internaljails>

pass in proto tcp from 127.0.0.1 port 25 to port 25

## INBOUND

########### ICMP4/6 ###########

# ICMPv4
pass inet proto icmp icmp-type $icmp_types

# ICMPv6
#pass on $ext_if inet6 proto icmp6 icmp6-type $icmp6_types
pass inet6 proto icmp6 icmp6-type $icmp6_types

########### Web Servers ###########

pass proto tcp to $nextcloud port $webports $tcp_state
pass proto tcp to $githost port $webports $tcp_state

########## Nextcloud ############

############# PLEX ##############
pass proto tcp from any to $plex port $plex_ports_tcp $tcp_state
pass proto udp from any to $plex port $plex_ports_udp $udp_state

############## SSH ##############

# Allow SSH in to jails that allow sshjails
pass in on $ext_if proto { tcp, udp } from 192.168.10.0/24 to <sshjails> port ssh $tcp_state

# Allow SSH into main host
pass quick proto { tcp, udp } from any to 192.168.10.40 port ssh $tcp_state

pass inet6 proto udp from any to any port $udp6_services $udp_state
pass inet proto udp from any to any port $udp_services $udp_state


##### OUTBOUND

pass out proto tcp to any port $webports
pass out proto tcp to any port 22

And how my networking is configured in my rc.conf

Code:
ifconfig_em0="DHCP"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"
 
You cannot* filter bridged traffic. pf filters on layer 3 and up, not layer 2.

* Not actually true. You can, with net.link.bridge.pfil_bridge, but it's a terrible idea. Because pf filters layer 3 and up. Do not be tempted to try to use that. It will blow up in your face and when you report a bug to me about it I will point and laugh.
 
Hmmm. I should have known. What's recommended then? I see quite a few sources saying they'd use NAT, and I also see some places saying you can do firewalling from within the jail? Maybe that would be a better idea. I thought I saw that IPFW works well inside a jail, and I wouldn't have to use any dev rules to pass any pf devices to any of the jails.
 
strange. I have this setup working for quite some time now with absolutely no issues. I block everything, and communication between jails is enabled via a line
Code:
pass quick inet proto tcp from 192.168.3.80 to 192.168.3.33 port 3306
on the host. In the jail the administrators can further narrow down their access with PF. can't see why yours should not work, hope I can have another look the next days
 
out of curiosity I just tinkered with a virgin system. bridge1 with interfaces igb0, epair0a, epair1a. Jail "test1" with IP 192.168.1.236 and jail "test2" with IP 192.168.1.237 (both are jails with standardtools, NO iocage). my pf.conf:
Code:
block out quick proto tcp from 192.168.1.237 to 192.168.1.155 port 22
pass quick proto tcp from 192.168.1.236 to 192.168.1.237 port 22
block out quick proto tcp from 192.168.1.236 to any
pass

works as expected.

side note: I switched from iocage to standard jails because of some issues (and this smells like one of that category... I vaguelly remember a problem with VNET really made me mad so I decided to switch immediately)
 
Hmmm. I should have known. What's recommended then? I see quite a few sources saying they'd use NAT, and I also see some places saying you can do firewalling from within the jail? Maybe that would be a better idea. I thought I saw that IPFW works well inside a jail, and I wouldn't have to use any dev rules to pass any pf devices to any of the jails.
You can route to your jails, and you'll be able to filter that traffic, yes.

Depending on your network setup you may need to do NAT, but if so that'll almost certainly be on the host and not in the jails.
Both pf and ipfw (and ipf) can be used inside vnet jails.
 
I did see that PF can be used in the jails, but I also saw something about having to pass through devices to the jails. Maybe not so bad anyways. If I recall, I saw that wasn't necessary with IPFW. I think it would be a lot easier to Not do NAT...now that I think about it, I would like to avoid that. Maybe rootbert is onto something with iocage having problems.
 
Yes, to use pf in a vnet jail you need to twiddle /etc/defaults/devfs.rules to add a "add path pf unhide" line to the end. I'm looking at just committing that, because it makes life easier for everyone. IPFW doesn't need that because it communicates differently with the kernel.

If you don't want to NAT that may be possible depending on your network setup. Your choice of firewall isn't going to make any difference in that. That's also entirely unrelated to iocage.
 
So what then will be the best practice for firewalling between VNET jails? My goal majorly is I don't want my external jails to be able to access my local-only jails. Separate networks entirely? I could create a second bridge from one of my interfaces and connect it to a separate network from my router? Just add firewall rules to each jail and say that suffices? (which, wouldn't protect from root-level compromises in the jail)
 
So what then will be the best practice for firewalling between VNET jails? My goal majorly is I don't want my external jails to be able to access my local-only jails. Separate networks entirely? I could create a second bridge from one of my interfaces and connect it to a separate network from my router?
Yes, you could create a separate bridge (and dedicated subnet) for your local-only jails, and use the firewall to deny them access to the rest of the work (and the rest of the world access to them).

Just add firewall rules to each jail and say that suffices? (which, wouldn't protect from root-level compromises in the jail)
You'd want to set firewall rules on the host, for that. That would still protect from jail compromises.
 
You cannot* filter bridged traffic. pf filters on layer 3 and up, not layer 2.

* Not actually true. You can, with net.link.bridge.pfil_bridge, but it's a terrible idea. Because pf filters layer 3 and up. Do not be tempted to try to use that. It will blow up in your face and when you report a bug to me about it I will point and laugh.
what is wrong with net.link.bridge.pfil_bridge? I am using it for years and it works great, it's also enabled by default.
 
It breaks hilariously with IPv6 fragment reassembly, for one.

The fundamental problem is that it takes a hack-y shortcut to make non-L2 aware firewalls (like pf) work on L2 traffic. There are edge cases that just plain don't work and produce funny things like IPv6 packets without Ethernet headers.
 
I know about the problem, yes ;)

There is at least one bug about it as well, but I don't know the number off the top of my head.

I have no current plans or aspirations to do anything about it. It'd be a pretty serious re-engineering of how all of that works and short of a customer who really wants to see it it's not going to happen any time soon.
 
If only I wasn't a single sponge, I'd provide funding for it ;)

For IPFW, there are sysctl flags that can be set to have it work on L2 right? Would that be silly? Maybe doing an extra bridge is the best thing anyways. I think that's pretty standard practice anyways...just trying not to over engineer for a home server lol
 
Sure! I appreciate your help with all this. You've given me some things to mull over. I may do a combination of IPFW and figuring out how to do a separate bridge. I need to read up on having 2 interfaces on separate networks without dorking my server.
 
Back
Top