PF bhyve VM cannot access internet

fred974

fred974

Hi,

I have created a bhyve M using sysutils/cbsd and when I log in to my VM, I cannot fetch port, ping or anything related to the outside world.
I have created some pf pass rules but I don't know if they are correct.
Could anyone please advise?
/etc/pf.conf
Code: 
ExtIf    = "lagg0"
Jail_net = "10.47.1.0/24"

www_int   = "10.47.1.11"
www_ext   = "194.12.xx.xxx"
sophimail = "194.12.xx.xxx"
ext       = "194.12.xx.xxx"
bhyvetest  = "194.12.16.xxx"
# --- jails ---
nginx         = "10.47.1.11"
unifi         = "10.47.1.12"
webjail       = "10.47.1.14"
colocation    = "10.47.1.15"

webservices = "{80, 443}"
unifiservices = "{8080, 8443, 8843, 8880, 8080}"
mailservices = "{25, 993, 995, 465, 143, 587}"
netbios_tcp = "{135, 139, 445}"
netbios_udp = "{135, 139, 445}"
IPv4_icmp_types = "{ echoreq, unreach }"

table <TRUSTED> persist file "/etc/pf-files/trusted.pftable"
table <BANNEDZONE> persist file "/etc/pf-files/bannedzones.pftable"


### all incoming traffic on external interface is normalized and fragmented
### packets are reassembled.
scrub in on $ExtIf all fragment reassemble

nat on $ExtIf from $Jail_net to !$Jail_net -> 194.12.xx.xxx

# Reverse Proxy
# --- redirect http traffic to the internal web proxy server ---
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port http  -> $www_int port  http
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port https -> $www_int port  https

# Unifi Controler
# --- redirect unifi controler traffic to the unifi jail server ---
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8080  -> $unifi port  8080
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8443  -> $unifi port  8443
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8843  -> $unifi port  8843
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8880  -> $unifi port  8880
rdr on $ExtIf inet proto udp from  !($ExtIf) to $www_ext port 8080  -> $unifi port  8080

# Netdata
# --- redirect Netdata to the internal webjail server ---
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 19999 -> $webjail port  19999

# Anchors
rdr-anchor "openvpn"

rdr on $ExtIf inet proto tcp from !($ExtIf) to $ext port 1124 -> $nginx        port 22
rdr on $ExtIf inet proto tcp from !($ExtIf) to $ext port 1125 -> $unifi        port 22
rdr on $ExtIf inet proto tcp from !($ExtIf) to $ext port 1126 -> $colocation   port 22
rdr on $ExtIf inet proto tcp from !($ExtIf) to $ext port 1127 -> $webjail      port 22


### set a default deny everything policy.
block log all

### exercise antispoofing on the external interface, but add the local
### loopback interface as an exception, to prevent services utilizing the
### local loop from being blocked accidentally.
set skip on lo

antispoof for $ExtIf inet

### get rid quick of Internet noise like microsoft netbios service.
### This accounts to 80% of dropped traffic. We don't need to log this also
block in quick on $ExtIf proto tcp from any to any port $netbios_tcp
block in quick on $ExtIf proto udp from any to any port $netbios_udp

anchor "openvpn"

### Quick blocks
#block drop in log quick on $ExtIf from <BLOCKTEMP> to any
block drop in log quick on $ExtIf proto tcp from <BANNEDZONE> to any


### $ExtIf inbound

pass in log on $ExtIf inet proto tcp from 91.203.xx.xxx to $sophimail port 22
pass in log on $ExtIf inet proto tcp from 208.95.xx.xxx to any port 10050

# --- pass icmp echo
pass in log on $ExtIf inet proto icmp all icmp-type $IPv4_icmp_types

# --- pass incoming http/https traffic --
pass in log on $ExtIf inet proto tcp  from !($ExtIf)    to $www_int port $webservices
pass in log on $ExtIf inet proto tcp  from !($ExtIf)    to $ubiquiti port $unifiservices

# --- pass incoming ssh traffic --
pass in quick log on $ExtIf proto tcp from <TRUSTED> to $ExtIf          port 22222
#pass in quick log on $ExtIf proto tcp from any to $ExtIf               port 22222
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $nginx          port 22
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $unifi          port 22
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $colocation     port 22
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $webjail        port 22

# -- pass netdata traffic --
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $webjail     port 19999

# --- pass bhyve traffic ---
pass in quick log on $ExtIf proto tcp from !($ExtIf) to $bhyvetest
pass in quick log on $ExtIf proto udp from !($ExtIf) to $bhyvetest
pass in quick log on $ExtIf proto udp from $bhyvetest to any
pass in quick log on $ExtIf proto tcp from $bhyvetest to any
    
# --- pass incoming ftp traffic ---
#pass in log on $ExtIf inet proto tcp  from !($ExtIf) to $ftp_int port 21
#pass in log on $ExtIf inet proto tcp  from !($ExtIf) to $ftp_int port 49000:51000

# --- pass incoming mail traffic ---
pass in log on $ExtIf inet proto tcp from any to $sophimail port $mailservices
pass in log on $ExtIf inet proto tcp from any to $sophimail port $webservices

### $ExtIf outbound
pass out log on $ExtIf inet proto { tcp, udp, icmp } from any to any modulate state
Do I need to do a NAT or RDR rule in addition?
The VM is running ubuntu and I also tested a VM running FreeBSD
 
I only have this in pf.conf:

Code: 
ext_if="wlan0"
int_if="tap0"
nat on $ext_if from !($ext_if) -> ($ext_if:0)

Relevant lines in rc.conf:
Code: 
cloned_interfaces="tap0"
ifconfig_tap0="inet 192.168.200.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"

In sysctl.conf:
Code: 
net.link.tap.up_on_open=1

Works well.
 
This is all I had to add for allowing the VM traffic:

Code: 
pass in quick on vm-port from any to any
 
Interface groups are different than user groups. See ifconfig(8). I'm not familiar with cbsd so I can't say if it uses the same interface groups or not, it really should if it doesn't because interface groups make writing firewall rules so much easier.
 
You must log in or register to reply here.
Back
Top