PF allow a bridge0 bhyve vm to use DHCP


I'm bridging my test bhyve VMs out the my normal LAN, via igb0 on my desktop.
I evidently need *some* pf changes to allow DHCP to work, but I have no idea
what I'm missing. The normal host system has no trouble getting its own DHCP
IP of course, via the same interface. When I disable pf, the VM gets DHCP and
can access the internet. Tips?

Here's the config, at present the VM is off so not added to the bridge.

# ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether ac:1f:6b:15:2d:78
    inet6 fe80::ae1f:6bff:fe15:2d78%igb0 prefixlen 64 scopeid 0x1 
    inet netmask 0xffffff00 broadcast 
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether ac:1f:6b:15:2d:79
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
    inet netmask 0xff000000 
    groups: lo 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-public
    ether 02:37:62:ce:b3:00
    nd6 options=1<PERFORMNUD>
    groups: bridge 
    id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000

# pf.conf
# /etc/pf.conf
# macros
protocols       = "{ tcp, udp }"
# blocked_ports   = "{ syslog, epmd, amqp, couchdb }"
blocked_ports   = "{ syslog, amqp, couchdb }"
tcp_services    = "{ domain, http, https, smtp, 2200, couchdb, amqp, 1978, 3389, 6600, 9000 }"
udp_services    = "{ domain, 9993, 10001, 21027 }"
plex_ports    = "{ 32400, 1900, 3005, 5353, 8324, 32469, 32410, 32412, 32413, 32414 }"
martians        = "{,,  \
           ,,, \
           , }"

# interfaces
extl_if = "igb0"
intl_if = "lo0"
jail_if = "lo1"
hive_if = "bridge0"
# networks
internet = $extl_if:network
intl_net = $intl_if:network
jail_net = $jail_if:network
hive_net = $hive_if:network
zero_net = ""
local_net= ""

# limits
# bigger state tables help erlang receive sockets faster
set limit { states 200000, frags 40000, src-nodes 40000 }
set timeout { adaptive.start 180000, adaptive.end 200000 }

# trusted nets and devices
set skip on {  $intl_if, $jail_if, $hive_if }
set skip on { $zero_if }

# tables
table <badhosts> persist file "/etc/pf.blocklist"

# clean packets are happy packets
scrub in all

# jails are allowed outbound connections but not inbound
# these should be set up explicitly using spiped or similar
nat on $extl_if proto $protocols from   $jail_net to any -> ($extl_if:0)
nat on $extl_if proto $protocols from   $zero_net to any -> ($extl_if:0)
# plex

# block by default
block in log all

# but allow legit internal traffic
pass in  quick on $extl_if proto { udp }      from any to $extl_if port $udp_services
pass in  quick on $extl_if proto { tcp }      from any to $extl_if port $tcp_services
pass in  quick on $extl_if from $local_net to $extl_if
pass in  quick on $extl_if proto { udp, tcp } from $local_net to any port $plex_ports
pass in  quick on $extl_if proto { udp, tcp } from $local_net to any port $udp_services

# you shall not pass
block drop in  quick on $extl_if from $martians to any
block drop out quick on $extl_if from any to $martians
block drop in  quick on $extl_if proto { udp, tcp } from any to any port $blocked_ports

# handle script kiddies and other nasties on demand
block drop in  quick on $extl_if from <badhosts> to any

# o ye of little faith
# pass in log all
pass out all
I evidently need *some* pf changes to allow DHCP to work, but I have no idea what I'm missing.
And that's when tools like tcpdump(1) become extremely useful. It allows you to actually see the packets, so you can see which ones are working and which aren't.
Aah this is what I was looking for: and I guess I'm going to need something like this:

pass in  quick on $extl_if  inet proto udp from any port = bootps to any port = bootpc
pass out quick on $extl_if inet proto udp from any port = bootpc to any port = bootps

tcpdump is all well and good on a little network but without a service name or port to filter I was somewhat swamped with other broadcast garbage.
Progress of a sort, I can see the rejected packets, still not clear why they aren't being allowed through. Should I be using the bridge or tap interface name in my rule perhaps?

dch@wintermute ~> sudo tcpdump -i pflog0 -vvv -s 1500 '(port 67 or port 68)' 

15:45:37.115470 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 328) > [udp sum ok] BOOTP/DHCP, Request from 58:9c:fc:06:ce:d3 (oui Unknown), length 300, xid 0x6284c545, secs 52, Flags [none] (0x0000)
      Client-Ethernet-Address 58:9c:fc:06:ce:d3 (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Discover
        Client-ID Option 61, length 7: ether 58:9c:fc:06:ce:d3
        MSZ Option 57, length 2: 576
        Parameter-Request Option 55, length 7: 
          Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
          Domain-Name, BR, NTP
        Vendor-Class Option 60, length 3: "d-i"
        END Option 255, length 0
        PAD Option 0, length 0, occurs 29
120 packets captured
148 packets received by filter
0 packets dropped by kernel
This is the discover packet send from your Guest(58:9c:fc:06:ce:d3) that can't get out. Consider the bridge0 as virtual L2 switch. So first you need to allow the Interface that your bhyve Guest is using to enter into the bridge0 let's say that this is tap0 interface. Also when you are using packet filtering it's better to filter the members of the bridge rather the bridge interface itself.

For the test first you can pass all on traffic on tap0. And then to manipulate only the public facing interface igb0. You can do this by skiping the tap0 or pass all traffic on it.

WAN(igb0) -- Switch(Bridge0) -- bhyve guest(tap0)

pass quick on tap0 all
pass quick on igb0 inet proto tcp from any port 67:68 to any port 67:68 keep state flags S/SA
pass quick on igb0 inet proto udp from any port 67:68 to any port 67:68 keep state

# you may consider using something like this to keep the states
pass out on $extl_if proto tcp all modulate state flags S/SA
pass out on $extl_if proto { udp, icmp } all keep state

#Instead of your last line
pass all out
OK! I have a working config, it just needs now to be trimmed back to a *safe* config:

# /etc/pf.conf
# macros
dhcp        = "{ bootpc, bootps }"

# interfaces
extl_if = "igb0"
hive_if = "bridge0"
bridge = "{ tap0, bridge0, igb0 }"

block in log all
# but allow legit internal traffic
# dhcp for bridged bhyve instances
pass in  quick on $bridge proto udp from any port $dhcp to any port $dhcp
pass out quick on $bridge proto udp from any port $dhcp to any port $dhcp

I'm assuming only some of the $bridge macro is actually necessary.
As i said in my previous post don't use the bridge interface to apply packet filtering on it. It's best to use the member interfaces that are in this bridge.

When packet filtering is enabled, bridged packets will pass through the filter inbound on the originating interface on the bridge interface, and outbound on the appropriate interfaces. Either stage can be disabled. When direction of the packet flow is important, it is best to firewall on the member interfaces rather than the bridge itself.

The bridge has several configurable settings for passing non-IP and IP packets, and layer2 firewalling with ipfw(8). See if_bridge(4) for more information.