PF PF outbound rule on a bridge member interface did not stop packets

L

l00g33k

Dear Exports,

I have a puzzle on my hand. I have a network isolated from the Internet. The freeBSD computer has 4 Ethernet ports, but only 3 are involved in this puzzle while the 4th is only used to access the freeBSD. My basic goal is to send some of the multicast from the up stream Ethernet port to the first down stream Ethernet port, and the rest to the second down stream Ethernet port. In other words I want to split to multicast traffic to two different networks as the combined traffic load overload the receivers. I can't get it to work like I hope, and I have done a few experiementations. I will give more details below. Sorry for the long post.

The short story:

These PF rules don't work like I expect:

Code: 
pass in log on igb0 inet all flags S/SA allow-opts label "USER_RULE: WAN floating any to any"
block drop out log on igb2 inet proto udp from any to any port 1110 >< 1113 label "USER_RULE: OPT2 block all"
block drop out log on igb3 inet proto udp from any to any port 1110 >< 1113 label "USER_RULE: OPT3 block all"
pass out log on igb2 inet proto udp from any to any port = 1111 label "USER_RULE: OPT2 pass stream 1"
pass out log on igb2 inet proto udp from any to any port = 1112 label "USER_RULE: OPT2 pass stream 1"
block drop out log on igb3 inet proto udp from any to any port = 1111 label "USER_RULE: OPT3 block 1111"
block drop out log on igb3 inet proto udp from any to any port = 1112 label "USER_RULE: OPT3 block 1112"

I recognize that the blocks are redundant but I think they should not affect the outcome.

How I execute it and the results:

Code: 
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -f nolog.txt ; pfctl -F states ; tcpdump -c 4 -n -i bridge0 portrange 1111-1114 ; tcpdump -c 4 -n -i igb2 portrange 1111-1114 ; tcpdump -c 4 -n -i igb3 portrange 1111-1114
22 states cleared
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:22:11.135334 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.135399 IP 172.16.1.3.6519 > 239.4.6.23.1112: UDP, length 1164
22:22:11.135436 IP 172.16.1.3.6520 > 239.4.6.23.1113: UDP, length 1164
22:22:11.135471 IP 172.16.1.3.6521 > 239.4.6.23.1114: UDP, length 1164
4 packets captured
8 packets received by filter
0 packets dropped by kernel
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
22:22:11.148333 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.149333 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.150327 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.151328 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
4 packets captured
4 packets received by filter
0 packets dropped by kernel
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb3, link-type EN10MB (Ethernet), capture size 262144 bytes
22:22:11.163322 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.164324 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.165320 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
22:22:11.166319 IP 172.16.1.3.6518 > 239.4.6.23.1111: UDP, length 1164
4 packets captured
4 packets received by filter
0 packets dropped by kernel

I tried a few variations on the last 4 lines of the rules:

  • pass igb2 and then block igb2: no packets on either igb2 and igb3
  • pass igb3 and then block igb3: no packets on either igb2 and igb3
  • pass igb2 and then block igb3: packets show up on both igb2 and igb3
  • pass igb3 and then block igb2: packets show up on both igb2 and igb3
So it seems like filtering on bridge member interfaces doesn't always work: pass and then block on the same member interface works, but pass and then block on a different member interface doesn't work, despite what I read every where all said it should work.

Now of course I could have some things wrong else where or I could have a basic misunderstanding. Hence this post. Any pointer is much appreciated.

Here's the details:

First of all, let me describe the actors in the whole setup. I will skip some details and only describe what I think is relevant. I believe those details do not matter. There are 4 equipments in the setup: 1 sender, the freeBSD, and 2 receivers. What is being sent is a multicast packet to 239.4.6.23 port 1111-1114. This is almost like a RTP/RTCP/IPTV system but not quite. There is no RTCP and there is no IGMP. The sender simply is manually set to multicast to the multicast address 239.4.6.23 port 1111-1114. Wireshark confirms that the packets flow even when there is no receivers. Similarly the receivers are simply manually to listen to the multicase 239.4.6.23 port 1111 and 1112. When the packet flows, the receivers report receptions.

There are 4 Ethernet ports on freeBSD:

  • igb0: connected to the sender
  • igb1: connected to the control PC but is not involved in the test
  • igb2: receiver 1
  • igb3: receiver 2
The basic freeBSD sysem is flashed using pfSense from https://www.pfsense.org/. I believe pfSense is also not a factor because I have dropped down to shell and manually load PF using pfctl, as you will see. I'm also aware that pfSense may over write PF rules from time to time; I have repeated the tests with the same results.

A bridge has been setup in pfSense to combine igb0, igb2, and igb3, as reported by:

Code: 
[2.4.4-RELEASE][root@pfSense.localdomain]/root: ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:fb:89:7b:98:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
[2.4.4-RELEASE][root@pfSense.localdomain]/root:

[2.4.4-RELEASE][root@pfSense.localdomain]/root: sysctl net.link.bridge | sort
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.ipfw: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1
net.link.bridge.pfil_onlyip: 0
[2.4.4-RELEASE][root@pfSense.localdomain]/root:

All member interfaces use fixed IPv4 addresses.

The system:

Code: 
[2.4.4-RELEASE][root@pfSense.localdomain]/root: uname -a
FreeBSD pfSense.localdomain 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #2 b00c407ba5d(RELENG_2_4_4): Mon Nov 26 11:41:48 EST 2018     root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense  amd64

Thanks for reading!
 
Turned out adding 'no state' to the last pair of pass rules made it worked as expected:

Code: 
pass out log on igb2 inet proto udp from any to any port = 1111 no state label "USER_RULE: OPT2 pass stream 1"
pass out log on igb2 inet proto udp from any to any port = 1112 no state label "USER_RULE: OPT2 pass stream 1"

With 'no state' option, only the inbound packet state was kept:

Code: 
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -ss | grep 1111
igb0 udp 239.4.6.23:1111 <- 172.16.1.3:6518       NO_TRAFFIC:SINGLE

Without 'no state' option, the inbound and one of the outbound (the wanted) packet states were kept:

Code: 
[2.4.4-RELEASE][root@pfSense.localdomain]/root: pfctl -ss | grep 1111
igb0 udp 239.4.6.23:1111 <- 172.16.1.3:6518       NO_TRAFFIC:SINGLE
igb2 udp 172.16.1.3:6518 -> 239.4.6.23:1111       SINGLE:NO_TRAFFIC

It's not clear to me why not having a state entry and having a rule to block does not prevent the packet from going out igb3. But at least my problem is no longer and it's just an understanding problem.

Thanks for reading.
 
I have a thought. It should be possible to write a simulator. Capture the packets on both interface without PF, then simulate the effect of the PF. Could be useful.
 
For those who used pfSense (which is where I come from,) make sure the WAN side pass any to any rule does not select any interface. Further more, for all the inbound pass rules and outbound block rules, make sure 'quick' is not selected, and 'State type' is 'None' (i.e. translates to 'no state' PF option.)
 
You must log in or register to reply here.
Back
Top