PF Bhyve and pf

Steffen

Steffen

Hello!

I have a debian guest running on FreeBSD 13.0 using bhyve.

If I disable pf the guest can communicate with WAN. A httpd running on debian is also reachable from WAN.

If I enable pf I can ssh into the debian guest, but the guest cannot communicate with WAN, neither is the httpd reachable.

I tried to debug with tcpdump but I cannot see any traffic on the bhyve host coming from the guest...

My interfaces:

Code: 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
    ether a4:bb:6d:xx:xx:xx
    inet 192.168.s.t netmask 0xffffff00 broadcast 192.168.s.x
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Code: 
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 58:9c:fc:xx:xx:xx
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1667

Code: 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:f4
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

I added for testing purposes this to pf.conf. I think the bridge rules aren't necessary but for God's sake...:

Code: 
# DEBUG: RULES FOR VMM
pass in quick on bridge0 all keep state
pass in quick on tap0 all keep state
pass out quick on bridge0 all keep state
pass out quick on tap0 all keep state


Full pf file:

Code: 
### INTERFACES ###
if = "{ lo0, em0 }"
vif = "{ ue0 }"

### SETTINGS ###
set block-policy drop

### OFFENE TCP/UDP-PORTS ###
tcp_pass = "{ SOME_TCP_POSTS }"
udp_pass = "{ SOME_UDP_PORTS }"
icmp_types = "echoreq"


### NORMALISATION ###
scrub in all
antispoof for $if


### TABLES ###
table <intranet>  { SOME_SUBNET_IP_RANGES }
table <smartnet>  { A_SUBNET}
table <whitelist> { SOME_ADMIN_MACHINES }
table <bruteforce> persist
table <blocklist> persist file "A_FILE_PATH"
table <ssh_allowed> persist

### RULES ###
set skip on lo0
block all
block quick from <bruteforce>


# DEBUG: RULES FOR VMM

pass in quick on bridge0 all keep state
pass in quick on tap0 all keep state
pass out quick on bridge0 all keep state
pass out quick on tap0 all keep state


pass in quick proto tcp from <whitelist> to any keep state
pass in quick proto udp from <whitelist> to any keep state

pass in quick proto tcp from <intranet> to any port 8443 keep state
pass in quick proto tcp from <intranet> to 192.168.x.x port 443 keep state
pass in on $if proto udp from <intranet> to any port $udp_pass keep state

pass in on $vif proto udp from <smartnet> to 192.168.s.t port 53 keep state
pass in on $vif proto tcp from <smartnet> to 192.168.s.p port 53 keep state
pass in on $vif proto tcp from 192.168.10.16 to 192.168.s.r port 8443 keep state

pass in log (all, to pflog0) on $if proto tcp from any to any port $tcp_pass flags S/SA keep state (max-src-conn 200, max-src-conn-rate 25/5, overload <bruteforce> flush global)
pass in on $if proto udp to any port $udp_pass keep state
pass out quick all keep state

# PING #
pass in on $if inet proto icmp all icmp-type $icmp_types keep state

# TRACEROUTE #
pass in on $if inet proto udp from any to any port 33433 >< 33626 keep state

Do you have any hints?

Thanks!
 
Starting with the basics first - I know, it's stupid, but do you have net.inet.ip.forwarding=1?

Do you see anything of relevance when tcpdump -i pflog0 -v?

Maybe you want to add some "block log" to get more details where the block occurs.
 
Code: 
[root@HOST ~]# sysctl -a | grep net.inet.ip.forward
net.inet.ip.forwarding: 1
[root@HOST ~]#

Code: 
tcpdump -i pflog0 -v
shows absolutely nothing, related to the debian guest. I added
Code: 
log
to the debug rules mentioned above, also no traffic.
 
Uhm, not sure - you let your traffic move between bridge and tap0, but since you've got your IP on your em0 device instead of bridge and I don't know your IP ranges, it's really hard to say, what's going on.

My take is - it's likely being blocked on your em0 interface.

You could tag your traffic coming from your VM and then just let it pass on em0. There's a bunch of IP ranges and I'm having difficulty following, what's what, to be honest. Maybe your VM is in those ranges and supposedly already passes?
 
cmoerz said:
Uhm, not sure - you let your traffic move between bridge and tap0, but since you've got your IP on your em0 device instead of bridge and I don't know your IP ranges, it's really hard to say, what's going on.

My take is - it's likely being blocked on your em0 interface.

You could tag your traffic coming from your VM and then just let it pass on em0. There's a bunch of IP ranges and I'm having difficulty following, what's what, to be honest. Maybe your VM is in those ranges and supposedly already passes?
Click to expand...
It is in the same range like the host, but with its own interface. As I pass quick on this interface (tap0), I really have no glue where the error is...
 
I can relate; pf troubleshooting can be a pain in the neck.

The thing is - just because you pass quick on tap0 doesn't immediately mean it's not still getting blocked elsewhere. You need to consider the packet flow and route.

It's your physical em0, which is connected to your switch. I'd expect packets to still move over said em0 to reach your gateway. So, I suppose your packets move from tap0 to bridge0 to em0.

Aren't you blocking those packets on em0 or is that vm operating in i.e. the "intranet" IP range, since those packets are also coming "in" and moving "out" across em0?

Again: I might be misinterpreting your setup.
 
Set an IP address for your bridge/tape e.g. 192.168.200.1
Set an IP address for your BhyveVM(s) e.g. 192.168.200.2

Set static route in rc.conf for traffic to 192.168.200.0/X via gateway 192.168.200.1. X could be 24.

Check routing on host (netstat -arn) that the subnet is included.

Add Pf rules:
- nat for the subnet(192.168.200.0/X) an
- additional table like smartnet for the subnet with pass in/out rules. - In fact, you can do redirect so that traffic at specific ports go through.
 
Yes, port 53 is mostly required open for both tcp & udp. Your case seemed more than that since the host works well and bhyve, when pf is disabled.
I'm glad you resolved it.
 
You must log in or register to reply here.
Back
Top