PF pf and bridge(4)

The following examples illustrate some curiosities I've found when filtering traffic between bridge members:

Code:
# /etc/pf.conf
jailsrc = "10.0.0.2"
jaildst = "10.0.0.3"
jailsrc_if = "epair0a"
jaildst_if = "epair1a"

# ***** SCENARIO 1 - Block by Default *****

block all
# Both rules are required to pass traffic from jailsrc to jaildst
pass on bridge0            from $jailsrc to $jaildst
pass on $jailsrc_if        from $jailsrc to $jaildst

# Interestingly, it doesn't matter what we do on the destination jails interface
block quick on $jaildst_if from $jailsrc to $jaildst

# ***** SCENARIO 2 - Pass by Default *****

pass all
# As expected, each single one of the following will block the traffic
block on bridge0           from $jailsrc to $jaildst
block on $jailsrc_if       from $jailsrc to $jaildst

# But not this one, the interface for the destination jail is ignored
block quick on $jaildst_if from $jailsrc to $jaildst
This is the bridge setup:

Bash:
ifconfig bridge create name bridge0
ifconfig br0 ether 0a:00:0a:00:00:01
ifconfig br0 inet 10.0.0.1/24
ifconfig br0 addm epair0a
ifconfig br0 addm epair1a

Now here are my questions:
1. Why is $jaildst_if not filtered? Given that $jailsrc_if is filtered, this seems to be some strangely inconsistent behavior.
2. Why is $jailsrc_if filtered? Given that $jaildst_if is not filtered, this also seems to be strangely inconsistent.

Either way, I would be really grateful to learn what's going on here.
 
Okay, part of my puzzle seems to be answered in the PARAMETERS section of the pf.conf(5) man page:
A packet always comes in on, or goes out through, one interface.
Given this I would think as far as pf(4) is concerned there are always at most two interfaces involved - the incoming interface, where the packet arrives on the system, and the outgoing interface where the packet leaves the system (they may be the same).

However, it still seems to be a rather arbitrary choice to take the bridge member as incoming interface and the bridge itself as outgoing interface.

Does anyone know the rational behind this?
 
Also note that I do not consider pf bridge filtering to be a supported use case. I know of major bugs (IPv6 packet reassembly and refragmentation goes hilariously wrong for one thing). When it breaks do not expect your bug reports to be acted on.
 
Does this mean that the moment you need to add a bridge interface to your config, you should switch to a different firewall solution? Or does this statement only apply to the use of ... on bridge0 ... (meaning that non-interface related filtering is still okay)?
 
No, you can filter just fine on bridges on L3. That is, you can filter IP traffic in and out of a bridge, but if you're trying to filter between bridge member interfaces (i.e. on L2) you're going to run into issues.
 
No, you can filter just fine on bridges on L3. That is, you can filter IP traffic in and out of a bridge, but if you're trying to filter between bridge member interfaces (i.e. on L2) you're going to run into issues.
If it's just a matter of swapping out the name of a bridge with the name of its member (which could also be used directly if it wasn't in the bridge), at what point does it become an L2 rule?

if_bridge(4) says that only ARP/REVARP get through on L2 when net.link.bridge.pfil_onlyip=1, yet you can still filter L3 (i.e., IP addresses; not MAC addresses) between bridge members in pf.conf(5) with that enabled. Doesn't that imply that if_bridge(4) sends L3 traffic between members through pfil(9)?

Or maybe you mean that if_bridge(4) leaks packets on L2 outside of whatever L3 packets it sends to pfil(9)?
 
Okay, so something like the following would be safe to do?
Code:
# /etc/pf.conf
jailsrc = "10.0.0.2"
jaildst = "10.0.0.3"

# Prevent spoofing - Option 1 (preferred)
# sysctl net.link.bridge.pfil_member=0
antispoof quick for bridge0

# Prevent spoofing - Option 2
# sysctl net.link.bridge.pfil_member=1
block in quick on bridge0 from ! 10.0.0.0/8
block out quick on bridge0 to ! 10.0.0.0/8

# Filter rules
block all
pass from $jailsrc to $jaildst
Are there any downsides in disabling filtering on bridge members alltogether (sysctl net.link.bridge.pfil_member=0)?
 
If I do this, bridge members cannot be prevented from talking to each other.
If you want to block all traffic between specific bridge members, you can set them both to private:

Code:
ifconfig bridge0 private epair0a
ifconfig bridge0 private epair1a

This will even block L2 such as ARP. Note that this doesn't work if only one of them is private, and I don't think you can selectively allow traffic through.
 
I want to enforce a communication policy between various jails. For example the mail server should be able to access the LDAP server, but not vice versa.

My idea was to use a bridge and a firewall on the host to enforce that policy. Unfortunately it doesn't seem to be as straight forward as I thought.

I actually learned about the problems of PF with L2 filtering on bridge interfaces already some time ago. However, I also thought that wouldn't apply to me anyway. What I don't understand is why it my above examples (especially the last one) would count as L2 filtering. All I'm doing is pass from $src_ip to $dst_ip - what am I missing here?

For L2 filtering and to prevent jails from spoofing their address(es) I'm actually not using an epair but ng_eiface(4) nodes with an ng_bpf(4) node sandwiched in between.

If I really have to sysctl net.link.bridge.pfil_bridge=0 and sysctl net.link.bridge.pfil_member=0 for pf to behave predictibly, I don't see any viable solution other than to not use it.
 
I don't know the implementation details of pf (or any other firewall software for that matter). From the outside, every IP network is on top of layer 2. What makes a bridge interface special from the point of view of a firewall?

Regarding why I'm not just routing between my jails: appart from making pf happy, what would be the benefit? It seems more complex than switching and I don't really need (or want) to have the jails on different networks.
 
I don't know the implementation details of pf (or any other firewall software for that matter). From the outside, every IP network is on top of layer 2. What makes a bridge interface special from the point of view of a firewall?
Because you're switching between your jails and not routing. If you use pf to filter routed traffic you're fine. If you try to make it handle switched traffic you're going to run into issues.

pf can filter just fine on traffic that enters on a bridge, gets routed and then sent out on another (or even the same) bridge. It will blow up in your face if you try to make it filter switched traffic on that bridge. I don't know how else to explain this.
 
  • Thanks
Reactions: mms
Kristof Provost I am struggling with exactly the same questions as mms. I am currently experimenting with VNET jails using if_bridge and if_epair devices. Also, I am trying to follow your advice to never filter on switched traffic. Even when I set net.link.bridge.pfil_bridge=0 and net.link.bridge.pfil_member=0, I am still not sure how I can recognize filtering on switched traffic in my /etc/pf.conf. Could you show some examples of what is ok and what is not ok?

For context, I am now considering to drop bridging all together, to just rely on if_epair devices, as follows:
Code:
scrub in all

set skip on lo0

ext_if = em0

nat on $ext_if from epair1a:network to any -> $ext_if static-port

block log

pass in quick on epair1a inet proto icmp from epair1a:network to epair1a:network
pass in quick on epair1a inet proto icmp from epair1a:network to $ext_if:network
pass in quick on epair1a inet proto { udp, tcp } from epair1a:network to any port domain
pass in quick on epair1a inet proto tcp from epair1a:network to any port { http, https }

pass out quick on epair1a inet proto icmp from epair1a:network to epair1a:network

pass out on $ext_if
 
Because you're switching between your jails and not routing. If you use pf to filter routed traffic you're fine. If you try to make it handle switched traffic you're going to run into issues.

pf can filter just fine on traffic that enters on a bridge, gets routed and then sent out on another (or even the same) bridge. It will blow up in your face if you try to make it filter switched traffic on that bridge. I don't know how else to explain this.
In cases where filtering L3 traffic between two interfaces in the same bridge actually works, is that just an accident, or is that particular feature poorly implemented and thus shouldn't be used? Is that traffic just prefiltered prior to switching, or does it get routed during the filtering process?

Do you mean that the bridge doesn't always correctly assemble an L3 packet from L2 frames in order to pass it to the filter, and therefore some L3 traffic might bypass the filter?

If the source is a bridge member and the destination is an IP address of the bridge itself, is that also unsafe?

The wording in if_bridge(4) implies that filtering between interfaces in the same bridge can be done, but that could just be a misinterpretation.
 
In cases where filtering L3 traffic between two interfaces in the same bridge actually works, is that just an accident, or is that particular feature poorly implemented and thus shouldn't be used? Is that traffic just prefiltered prior to switching, or does it get routed during the filtering process?
No, that's fine. While not common it is a valid use case to route between logical networks on the same physical interface. That will work just as well as routing between two different physical interfaces.

Do you mean that the bridge doesn't always correctly assemble an L3 packet from L2 frames in order to pass it to the filter, and therefore some L3 traffic might bypass the filter?
First part, yes. It's not so much a question of bypassing the filter as it is of resulting in totally bogus packets being sent out. (As in: IPv6 packets without ethernet header being sent out.)
If the source is a bridge member and the destination is an IP address of the bridge itself, is that also unsafe?
No, anything L3 will work just as well on a bridge interface as on any other interface.
The wording in if_bridge(4) implies that filtering between interfaces in the same bridge can be done, but that could just be a misinterpretation.
It can, but shouldn't. There are landmines.
 
Back
Top