PF Filtering VNET jails from host side without using NAT?

K

kreyszig

Hi,

I am trying to set up a bunch of VNET jails and I want to filter the traffic between jails and between the jails and the host, from the host side. The jails and the host are on the same subnet and I do not want to use NAT. Currently I am bridging the ethernet interface with the epairs. It does work from a routing perspective, but it does not seem to be possible to filter the traffic to the jails using this technique, the traffic from either the host or other computers on the same subnet go unfiltered to the jails even if I try blocking all traffic. Any suggestion about how this could be done? I would like to filter the traffic on the host side. Thanks!
 
Anyone? This is what I have on the host side:
Code: 
dpni0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether hh:hh:hh:hh:hh:hh
        inet 192.168.1.240 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether hh:hh:hh:hh:hh:hh
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair241a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: dpni0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
epair241a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vnet-multimedia
        options=8<VLAN_MTU>
        ether 02:0a:fb:64:78:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Code: 
pfctl -s rules
block drop in log all
block drop out log all
pass inet proto tcp from 192.168.1.0/24 to 192.168.1.240 port = 22 flags S/SA keep state
pass out on dpni0 inet all flags S/SA keep state
anchor "multimedia" all

pfctl -a multimedia -s rules
block drop in log on epair241a all
block drop out log on epair241a all

In the jail I have this:
Code: 
epair241b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:0a:fb:64:78:0b
        inet 192.168.1.241 netmask 0xffffff00 broadcast 192.168.1.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Now if I try sshing the jail from an external machine I see this on the host side:
Code: 
tcpdump -i dpni0 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on dpni0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:57:21.896158 IP 192.168.1.139.41878 > 192.168.1.241.22: Flags [S], seq 3194247819, win 64240, options [mss 1460,sackOK,TS val 3219838790 ecr 0,nop,wscale 7], length 0[/S]


Code: 
tcpdump -i epair241a -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on epair241a, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:57:54.819571 IP 192.168.1.139.54708 > 192.168.1.241.22: Flags , seq 104903793, win 64240, options [mss 1460,sackOK,TS val 3219871694 ecr 0,nop,wscale 7], length 0
08:57:54.819615 IP 192.168.1.241.22 > 192.168.1.139.54708: Flags [R.], seq 0, ack 104903794, win 0, length 0


The traffic does not get filtered, "tcpdump -n -e -ttt -i pflog0" shows nothing from the host side
 
Ok, so it looks like I had to set net.link.bridge.pfil_member to 1 ... Now unless I want no filtering for the host IP, I have to open each port I need for the jail both on the external interface and on the jail's epair. Is it the best way to do it, or should I instead do something else like creating an epair for the host, bridge it with the other interfaces, pass in all inbound traffic on the external interface, and only pass out what is desired on any epair?
 
rootbert said:
I suggest avoid a bridge and use routed networking
Click to expand...
Thank you. Could you please give me an example of how this could be done without using NAT? After changing the sysctl variable, now the traffic outgoing from my external interface gets blocked even if I allow it to pass in pf. I was not too keen on using a bridge to begin with, but I was not sure how else to get it to work without NAT.
 
Seems like it's better to use a bridge for your case, forget NATing. So, assuming you are using a bridge and have pfil_member set, then what is wrong with the filter ruleset you posted above? You don't have to open ports on the host for all the ports you use on your jail, just allow the traffic to pass. For example, using your rule set above,

Code: 
pfctl -s rules
block drop in log all
block drop out log all
pass inet proto tcp from 192.168.1.0/24 to 192.168.1.240 port = 22 flags S/SA keep state
pass in on dpni0 inet proto tcp from 192.168.1.0/24 to 192.168.2.141 port = 80
pass out on dpni0 inet all flags S/SA keep state
anchor "multimedia" all

pfctl -a multimedia -s rules
block drop in log on epair241a all
pass in on epair241a inet proto tcp port = 80
block drop out log on epair241a all

The two additional pass rules pass traffic on port 80 through to your multimedia host without opening up "ports" on the primary host.
 
junkyalleycat said:
Seems like it's better to use a bridge for your case, forget NATing. So, assuming you are using a bridge and have pfil_member set, then what is wrong with the filter ruleset you posted above? You don't have to open ports on the host for all the ports you use on your jail, just allow the traffic to pass. For example, using your rule set above,

Code: 
pfctl -s rules
block drop in log all
block drop out log all
pass inet proto tcp from 192.168.1.0/24 to 192.168.1.240 port = 22 flags S/SA keep state
pass in on dpni0 inet proto tcp from 192.168.1.0/24 to 192.168.2.141 port = 80
pass out on dpni0 inet all flags S/SA keep state
anchor "multimedia" all

pfctl -a multimedia -s rules
block drop in log on epair241a all
pass in on epair241a inet proto tcp port = 80
block drop out log on epair241a all

The two additional pass rules pass traffic on port 80 through to your multimedia host without opening up "ports" on the primary host.
Click to expand...
Thanks. I think I am still missing something. Here is what I have:
pfctl -s rules
block drop log all
pass in on dpni0 inet proto tcp from 192.168.1.0/24 to 192.168.1.240 port = ssh flags S/SA keep state
pass out on dpni0 inet all flags S/SA keep state
anchor "multimedia" all
Click to expand...
pfctl -a multimedia -s rules
pass in on epair241a inet all flags S/SA keep state
pass out on epair241a inet proto tcp from 192.168.1.0/24 to any port = ssh flags S/SA keep state
pass in on dpni0 inet proto tcp from 192.168.1.0/24 to 192.168.1.241 port = ssh flags S/SA keep state
Click to expand...

When using this I am able to ssh 192.168.1.241 from another machine on the same LAN. However, when I try to ssh 192.168.1.240 from that same other machine, I see:
tcpdump -n -e -ttt -i pflog0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 262144 bytes
00:00:00.000000 rule 0/0(match): block out on dpni0: 192.168.1.240.22 > 192.168.1.139.56296: Flags [S.], seq 3519596586, ack 1887787688, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 4009808 ecr 3248887944], length 0
Click to expand...

What is going on, given that I have "pass out on dpni0 inet all flags S/SA keep state" ?

Thanks!
 
hmm, not sure offhand, are you sure that log message corresponds with your attempt? the state for the ssh connection to 1.240 is created when it passes the in rule, from then on the rules shouldn't be considered since state exists for the connection. have you checked your ssh log for connection attempts, have you tried tcpdump against dpni0 to see what the traffic flow looks like there? You could also dump pf state (pfctl -ss) and check to see if you have state for that rejected packet, if the connection wasn't established it should be in a close_wait state or something (check quickly though as the state will go away). this will at least give you data points on whether the state was established or not. What are your pf.conf contents?
 
junkyalleycat said:
hmm, not sure offhand, are you sure that log message corresponds with your attempt? the state for the ssh connection to 1.240 is created when it passes the in rule, from then on the rules shouldn't be considered since state exists for the connection. have you checked your ssh log for connection attempts, have you tried tcpdump against dpni0 to see what the traffic flow looks like there? You could also dump pf state (pfctl -ss) and check to see if you have state for that rejected packet, if the connection wasn't established it should be in a close_wait state or something (check quickly though as the state will go away). this will at least give you data points on whether the state was established or not. What are your pf.conf contents?
Click to expand...
Yes I am sure it is my attempt, I tried multiple times. When I tcpdump dpni0 I get:
19:17:43.118543 IP 192.168.1.139.59574 > 192.168.1.240.22: Flags , seq 3828860690, win 64240, options [mss 1460,sackOK,TS val 3268031888 ecr 0,nop,wscale 7], length 0
Click to expand...

"pfctl -s s" returns nothing

My pf.conf is the following:
ext_if = "dpni0"

local_tcp_services = "{ 22 }"
localnet = $ext_if:network
host = 192.168.1.240

set skip on lo0
set block-policy drop

block log all

pass out on $ext_if inet to any keep state
pass in on $ext_if inet proto tcp from 192.168.1.0/24 to $host port $local_tcp_services flags S/SA keep state
anchor multimedia
Click to expand...

I have a pf_multimedia.conf that contains
ext_if = "dpni0"
vnet_ext_if = "epair241a"
vnet_ip = 192.168.1.241
local_tcp_services = "{ 22 }"
localnet = $ext_if:network

pass in on $vnet_ext_if inet to any keep state

pass in on $ext_if inet proto tcp from $localnet to $vnet_ip port $local_tcp_services flags S/SA keep state
pass out on $vnet_ext_if inet proto tcp from $localnet to any port $local_tcp_services flags S/SA keep state
Click to expand...

Note: I can ssh 1.241 just fine, it is only the 1.240 that does not work, like if pf was behaving differently when traffic whose source IP is the external interface's IP is attempting to come out from this external interface which is bridged with the epair.
 
So I think I got it to work, but I don't know why this works and not the other method. I now configure the IP on the bridge instead of the external interface. So I have
cloned_interfaces="bridge0"
ifconfig_bridge0="addm dpni0 SYNCDHCP"
ifconfig_dpni0="up -tso -vlanhwtso"
Click to expand...
instead of
ifconfig_dpni0="DHCP up -tso -vlanhwtso"
cloned_interfaces="bridge0"
create_args_bridge0="addm dpni0"
Click to expand...
Strangely, while I have to filter the ssh port for the host using the bridge interface,
block drop log all
pass out on bridge0 inet all flags S/SA keep state
pass in on bridge0 inet proto tcp from 192.168.1.0/24 to 192.168.1.240 port = ssh flags S/SA keep state
anchor "multimedia" all
Click to expand...
I have to filter it for the jail using the external interface:
pass in on epair241a inet all flags S/SA keep state
pass out on epair241a inet proto tcp from 192.168.1.0/24 to any port = ssh flags S/SA keep state
pass in on dpni0 inet proto tcp from 192.168.1.0/24 to 192.168.1.241 port = ssh flags S/SA keep state
Click to expand...
Anything else I tried did not work. If someone could explain it to me that would be greatly appreciated. I assume it is due to some implementation oddity? Thanks!
 
oh nice catch, i've always read that the ip should be bound to the bridge, i think it might work otherwise if you set net.link.bridge.pfil_local_phys, but i've never tried, you can read the bridge manpage about phys and the section about mac confusion and it might give you more reasons why. i've also had issues with multicast not working if the host ip is not bound to the bridge.

btw, you can use () to capture the ip of an interface so you don't have to encode them into your pf.conf, for example, (dpni0) would refer to 192.168.1.240, and (dpni0:network) would refer to 192.168.1.0/24, it can make writing the pf.conf a bit cleaner and remove some of the magic placeholders from it. check the pf.conf man page for more information on these sort of replacements.

also about filtering, for what it's worth, i don't filter on my bridge at all, filter only on the "physical" interfaces, i set my bridge to skip in the pf.conf.
 
junkyalleycat said:
oh nice catch, i've always read that the ip should be bound to the bridge, i think it might work otherwise if you set net.link.bridge.pfil_local_phys, but i've never tried, you can read the bridge manpage about phys and the section about mac confusion and it might give you more reasons why. i've also had issues with multicast not working if the host ip is not bound to the bridge.

btw, you can use () to capture the ip of an interface so you don't have to encode them into your pf.conf, for example, (dpni0) would refer to 192.168.1.240, and (dpni0:network) would refer to 192.168.1.0/24, it can make writing the pf.conf a bit cleaner and remove some of the magic placeholders from it. check the pf.conf man page for more information on these sort of replacements.

also about filtering, for what it's worth, i don't filter on my bridge at all, filter only on the "physical" interfaces, i set my bridge to skip in the pf.conf.
Click to expand...
Thanks,

For the host side, it does not work if I don't filter on the bridge, in pflog the traffic to 192.168.1.240:22 only shows up on bridge0 and not on dpni0, this is the strange behaviour I was referring to in my previous message. For jails it is the other way around, the traffic to 192.168.1.241:22 only shows up on dpni0 in pflog, and not on bridge0!
 
You must log in or register to reply here.
Back
Top