PF Working example for accessing a bhyve VM from the Internet

odhiambo

odhiambo

I am using PF on FreeBSD 13.2.
I have installed a VM using bhyve and I'd like to be able to access certain services running inside the VM.
From the host, I am able to access the service ports in a VM (ssh, HTTP, etc).
I am stuck on how to adjust my pf.conf so that I can access SSH/HTTP from the Internet.
I have two rules:
Code: 
# VM1 HTTP access
rdr pass on $ext_if inet proto tcp from any to any port { 8082, 8999 } \
        -> 172.16.0.99 port 80
# VM SSH access
rdr pass on $ext_if inet proto tcp from any to any port 2222 \
        -> 172.16.0.99 port 22

Both of these rules don't work.

My sysctl.conf:
Code: 
net.inet.ip.forwarding=1
net.link.tap.up_on_open=1
net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_local_phys=1

In my /etc/rc.conf:
Code: 
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3 tap4 tap5"
ifconfig_bridge0_name="vmbridge"
ifconfig_vmbridge="addm em1 addm tap0 addm tap1 addm tap2 addm tap3 addm tap4 addm tap5 up"
ifconfig_vmbridge_alias0="inet 172.16.0.1 netmask 255.255.255.0"

My full pf.conf can be seen at: https://pastebin.ubuntu.com/p/cWxCdNxFGH/

Thank you in advance for any insights.
 
Your VMs are directly attached to the 172.16.0.0/24 network on em1. No need for PF. You can access those ports directly on the IP addresses of the VMs.
 
SirDice thanks for the response.
There is something I am unable to wrap my mind around.
You say that my VMs are directly attached to the 172.16.0.0/24 network on em1. I suppose that's because of the bridging, right?
If I `telnet HOST 8082`, nothing happens and I get a timeout.
I need PF to help me in doing the NAT for that to work, no? Suppose I disabled PF, how will the HOST even handle that port since nothing is listening on it?
 
odhiambo said:
You say that my VMs are directly attached to the 172.16.0.0/24 network on em1. I suppose that's because of the bridging, right?
Click to expand...
Correct.
odhiambo said:
If I `telnet HOST 8082`, nothing happens and I get a timeout.
Click to expand...
From where are you doing this?

odhiambo said:
I need PF to help me in doing the NAT for that to work, no?
Click to expand...
There is no NAT involved on the host. Your VMs are directly bridged onto the network via em1.
 
My Host is in a Data Centre some 15km away from me. I am trying to connect to it from home.
My VM with IP 172.16.0.99 (Ubuntu) has both SSH and HTTP services enabled. I'd like to be able to access these services from anywhere on the Internet.
 
odhiambo said:
I am trying to connect to it from home.
Click to expand...
Your host is connected to a private range network. So you probably have a firewall that's in between the internet and this host. You need to open up ports and do some redirections there.

Can you make a network diagram? Your pf.conf is super confusing and doesn't give a clear picture on how things are connected.
 
My Host has two NICs, em1 and em0.
em1 is connected to the Internet via a public IP. This is the only active interface.
em0 is configured with 192.168.55.x, but it does nothing really. It's not connected to anything. I could remove all references to it from the pf.conf.
So all services on this server as as good as running on the host itself.
Does that paint a clearer picture of the network?

Thanks in advance.
 

Attachments

  • 1697717142713.jpg
    1697717142713.jpg
    124.2 KB · Views: 59
Ok, that takes away some of the confusion. How's the host connected to the internet? Because em1 has a private range address, so it's not directly connected to the internet. If em1 has the actual, real, internet address then you shouldn't add that interface to the bridge. A bridge(4) is a layer 2 connection, it's similar to what a physical switch does.
 
The private range of IP addresses are specified in the vmbridge as aliases.

```
cloned_interfaces="bridge0 tap0 tap1 tap2 tap3 tap4 tap5"
ifconfig_bridge0_name="vmbridge"
ifconfig_vmbridge="addm em1 addm tap0 addm tap1 addm tap2 addm tap3 addm tap4 addm tap5 up"
ifconfig_vmbridge_alias0="inet 172.16.0.1 netmask 255.255.255.0"
```

[19:54 ~ ]$ ifconfig em1
em1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 74:XX:XX:XX:XX:XX
inet 41.212.yy.xx netmask 0xffffff00 broadcast 41.212.yy.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[19:54 ~ ]$ ifconfig vmbridge
vmbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether yy:xx:nn:bb:df:1d
inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 2000000
groups: bridge
nd6 options=1<PERFORMNUD>
 
Hello SirDice , interesting reply above, and I wanted to learn;

When host machine has only a single, physical interface like "re0" with a public IP address, and in case if I need to have few VMs (creating a virtual 192.168.1.1/24 IP block for them), and, via PF I manage NAT and some port forwarding from them to open them to public internet, shouldn't I add my host's re0 interface to the bridge?

I created my switch like:

vm switch create -a 192.168.1.1/24 public
vm switch add public re0

So, I shouldn't vm switch add public re0 at all? Why?

Thanks.
 
kalleboy said:
shouldn't I add my host's re0 interface to the bridge?
Click to expand...
You should not, no.
kalleboy said:
Why?
Click to expand...
Because you want to control movement of packets between your internal network and an external (internet) network. One connection goes to the big bad network, the other is tied to your LAN. In this case your "LAN" is the internal network on the bridge.

[PC1]-----[router]-----[PC2]
PC1 and PC2 are on two separate networks.

[PC1]----[bridge/switch]----[PC2]
PC1 and PC2 are on the same network.
 
You must log in or register to reply here.
Back
Top