bhyve No Network Connection From bhyve Guest With PF Enabled

brianjm

New Member

Reaction score: 3
Messages: 18

EDIT: 17MAR2021

I would like to make a quick note showing how I solved this. I actually noticed my mistake after reading reading this forum post.

I never allowed my vm-bhyve interface vm-public through my firewall. Because my default setting is to simply block everything, my VMs weren't getting through. Adding these lines to my config changed that:

Code:
pass in quick on vm-public all
pass out quick on vm-public all

It could probably do with some fine tuning, but it is a relief to have solved this.



ORIGINAL POST:

I am having difficulty getting network connections to work from within my bhyve guest (managed with vm-bhyve) while my PF firewall is running. When I disable or stop PF, my VM guest has no issues connecting to the world, and I have no issues connecting to it. But as soon as PF starts running, I lose all connectivity to and from the VM.

My vm-bhyve info:

Code:
root@server:/zroot/vm # vm list
NAME    DATASTORE  LOADER  CPU  MEMORY  VNC           AUTOSTART  STATE
fedora  default    uefi    2    2G      0.0.0.0:5900  No         Running (29573)


root@server:/zroot/vm # vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     em0


root@server:/zroot/vm # vm switch info public
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: em0
  bytes-in: 206064422 (196.518M)
  bytes-out: 2696376408 (2.511G)

  virtual-port
    device: tap0
    vm: fedora

My output of ifconfig:
Code:
root@server:/zroot/vm # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 44:39:c4:39:99:f8
        inet 192.168.1.19 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.103 netmask 0xffffffff broadcast 192.168.1.103
        inet 192.168.1.101 netmask 0xffffffff broadcast 192.168.1.101
        inet 192.168.1.100 netmask 0xffffffff broadcast 192.168.1.100
        inet 192.168.1.102 netmask 0xffffffff broadcast 192.168.1.102
        inet6 fe80::4639:c4ff:fe39:99f8%em0 prefixlen 64 scopeid 0x1
        inet6 fd00::4639:c4ff:fe39:99f8 prefixlen 64 autoconf
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 6e:f7:f2:98:fd:0d
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-fedora-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:89
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 29573

And my PF config (I'm sure my PF config could use some work too, still learning how to put one together):

Code:
root@server:/zroot/vm # cat /etc/pf.conf
ext_if = "em0"
lo_if = "lo0"
icmp_types = "{ echoreq unreach }"
table <bruteforce> persist
table <webcrawler> persist
table <rfc6890> { 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16          \
                  172.16.0.0/12 192.0.0.0/24 192.0.0.0/29 192.0.2.0/24 192.88.99.0/24    \
                                  192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24            \
                                  240.0.0.0/4 255.255.255.255/32 }

# custom blacklist file
table <ipblacklist> persist file "/etc/ipblacklist"

# Set logging interface
set loginterface $ext_if

# Skip loopback interface
set skip on $lo_if

set skip on bridge0

# Sanitize traffic
scrub in all fragment reassemble max-mss 1440
                                                                                                                                                                                                                                                        [0/270]
# Set default polic[ies]y                                                                                                    
block drop in log all                                        
antispoof for $ext_if inet                                    
block quick from <ipblacklist>                                                                
                                                                                                                             
# Block all ports in the rfc6890 table to mitigate DDoS attacks                                                              
block in quick on egress from <rfc6890>                      
block return out quick on egress to <rfc6890>                                                         
                                                             
# Block spoofed packets                                      
antispoof quick for $ext_if                                                                                                  
                                                                                                                             
# Allow essential outgoing traffic                            
pass out quick on $ext_if proto tcp to any                    
                                                             
# Allow SSH. Limits number of simultaneous hosts from a single connection                                                    
pass in on $ext_if proto tcp to port { 22 } \                                                                                
        keep state (max-src-conn 15, max-src-conn-rate 3/1, \
                overload <bruteforce> flush global)          
                                                             
# Settings for a webserver                                    
pass in on $ext_if proto tcp to port { 80 443 } \            
        keep state (max-src-conn 45, max-src-conn-rate 9/1, \
                overload <webcrawlers> flush global)
                                                             
# Allow for samba, ftp, and other services to communicate
pass in on $ext_if proto tcp from any to any port { 445 137 138 139 ftp 3000 nfsd ntp 5900 9090 } keep state                  
pass in on $ext_if proto udp from any to any port { 445 137 138 139 ftp 3000 nfsd ntp 5900 9090 } keep state                  
                                                           
# Allow for a few other services to communicate with outside  
pass out proto { tcp udp } to port { 22 53 80 123 443 }

# Allow for the ping utility
pass inet proto icmp icmp-type $icmp_types

I have added ports 5900 and 9090 which are for VNC and the Cockpit web interface (running on my Fedora 33 guest) respectively. I have cut out some of the irrelevant parts like my jail settings. I tried adding a pass in on bridge0 and a skip for the same thing with no success.
 
Last edited:

richardtoohey2

Well-Known Member

Reaction score: 247
Messages: 455

I'm likely to be wrong - but 192.168.0.0/16 is 192.168.0.0 to 192.168.255.255 - and looks like you are using 192.168.1.x? Or will that rule not apply?

Could you try knocking the rfc6890 rules and see if it makes any difference?
 
OP
B

brianjm

New Member

Reaction score: 3
Messages: 18

I'm likely to be wrong - but 192.168.0.0/16 is 192.168.0.0 to 192.168.255.255 - and looks like you are using 192.168.1.x? Or will that rule not apply?

Could you try knocking the rfc6890 rules and see if it makes any difference?
My whole network is on 192.168.1.X. My router is 192.168.1.1, the VM host is 192.168.1.19, my jails are 192.168.1.100-106. And commenting out the rfc6890 rule set didn't have any effect, still no network connection on the VM.

When I SSH into the VM or use my VNC viewer (Remmina), I can't get any network connection to the outside:

Trying to ping my router from while SSH'd into the guest:

Code:
[guest]$ ping -c3 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms
 

richardtoohey2

Well-Known Member

Reaction score: 247
Messages: 455

Hopefully someone who knows more about this will come along soon.

While you are waiting you could try going down to the smallest pf configuration that works - even if a line or two of rules. Then add another rule, retry pf, another rule, retry pf - until you find out what rules are blocking your traffic or causing the issue.

Before you try that - you've got some logging - if you add more, any clues in there when you watch the logging?

I usually fix my pf problems with logging or stripping down to nothing and slowly building up the rules until I find where I was going wrong.
 
Top