Solved Bhyve assign public /29 subnet addresses to VM's

Hello everyone

So I have a dedicated server with Hetzner running FreeBSD 12.1-RELEASE-p3 currently not running PF firewall.
I have a /29 block of addresses assigned to me and I am trying to use them for Bhyve VM's but I am unsure how to go about it.

I have read the following form threads before posting:

I followed the instructions from the Hetzner wiki on FreeBSD addressing and can confirm that the address work when configured in /etc/rc.conf - That is they all route to the dedicated server as expected.

From my understanding of reading the above FreeBSD threads to use public IP addresses in Bhyve I have to remove the related ifconfig entries from /etc/rc.conf
Code:
ifconfig_em0_alias0="inet xx.xx.xx.xxx netmask xxx.xxx.xxx.xxx"
#additional entries omitted for brevity

So I commented out the ifconfig entries from /etc/rc.conf and configured Bhyve using the following guide.
I used the following commands in order.
Code:
1. pkg install vm-bhyve grub2-bhyve
2. zfs create dn42/bhyve
3. sysrc vm_enable="YES"
4. sysrc vm_dir="zfs:/dn42/bhyve"
5. vm init
6. cp /usr/local/share/examples/vm-bhyve/* /dn42/bhyve/.templates/
7. vm switch create public
8. vm switch add public em0
9. vm iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.3.0-amd64-netinst.iso
10. vm create lonniemason
11. vm install lonniemason debian-10.3.0-amd64-netinst.iso
12. vm console lonniemason

The contents of /dn42/bhyve/lonniemason/lonniemason.conf are:
Code:
loader="grub"
cpu=1
memory=1024M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="lmsystem.img"
grub_run_partition="1"
grub_run_dir="/boot/grub"
uuid="45ff5cbd-8f24-11ea-81ab-60a44c22a490"
network0_mac="58:9c:fc:00:bd:0d"

Output of vm switch list
Code:
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     em0
Output of vm switch info public
Code:
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: em0
  bytes-in: 2009995 (1.916M)
  bytes-out: 3222092 (3.072M)

  virtual-port
    device: tap0
    vm: lonniemason

So I proceeded with the installation of Debian knowing that this is not going to work because without the related ifconfig entries in /etc/rc.conf the FreeBSD system is unaware of the /29 Subnet.
If FreeBSD is unaware of the /29 Subnet then how is Bhyve expected to be aware of the /29 Subnet and how to route it.

Set IP Address:
Debian IP.png

Set Netmask:
Netmask.png

Set gateway:
Debian Gateway.png

Unable to reach gateway as expected:
Unreachable gateway.png


The /29 block of addresses assigned to me by Hetzner uses my main dedicated server IP as the default gateway, even though the gateway address appears to be in a different subnet - not sure how that works.

The current contents of my /etc/rc.conf are
Code:
hostname="lonniemason"
keymap="uk.kbd"
ifconfig_em0="inet 46.4.50.x netmask 255.255.255.192"
defaultrouter="46.4.50.1"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
gateway_enable="YES"
vm_enable="YES"
vm_dir="zfs:dn42/bhyve"
#ifconfig entries are commented out
#ifconfig_em0_alias0="inet 49.12.49.xxx netmask 255.255.255.248" #Network ID cannot be used by a host as an IP address?
#ifconfig_em0_alias1="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias2="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias3="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias4="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias5="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias6="inet 49.12.49.xxx netmask 255.255.255.255" #Usable address
#ifconfig_em0_alias7="inet 49.12.49.xxx netmask 255.255.255.255" #Broadcast addressxxx

Output of ifconfig
Code:
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 60:a4:4c:22:a4:90
    inet 46.4.50.x netmask 0xffffffc0 broadcast 46.4.50.xx
    inet6 fe80::62a4:4cff:fe22:a490%em0 prefixlen 64 scopeid 0x1
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 4a:68:0d:fa:ab:bb
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge vm-switch viid-4c918@
    nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-lonniemason-0-public
    options=80000<LINKSTATE>
    ether 00:bd:97:b4:02:00
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1683

Output of netstat -r
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            static.1.50.4.46.c UGS         em0
46.4.50.0/26       link#1             U           em0
dn42.lonniemason.n link#1             UHS         lo0
localhost          link#2             UH          lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              localhost          UGRS        lo0
localhost          link#2             UH          lo0
::ffff:0.0.0.0/96  localhost          UGRS        lo0
fe80::/10          localhost          UGRS        lo0
fe80::%em0/64      link#1             U           em0
fe80::62a4:4cff:fe link#1             UHS         lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
ff02::/16          localhost          UGRS        lo0

So at the very least I suspect that this is related to routing i.e somehow Bhyve needs to be aware of the /29 address space and how to route it.
Additionaly I am not 100% sure if my Bhyve switch config is correct for the scenario that I am trying to achieve.
Also I should note that I am new to FreeBSD but have had a good look at the FreeBSD handbook.

Any help would be greatly appreciated thanks :)

Lonnie Mason
 
Your post is quite long and I would suggest that you formulate your question in a shorter and more concrete way.
With that said, I hope I understood your matter correctly.

So you have a physical server and a block of IP addresses associated with it and you want to map each address to a bhyve VM, right?
Basically it does not make much difference if you do it for VMs, jails or physical machines. Your host machine could act as a router (so you need to configure it to forward IP packets to the VMs) and each of the VMs could use the host as a gateway.
This handbook section is dedicated to gateways and routers: https://www.freebsd.org/doc/handbook/network-routing.html

One possible configuration is:
Each VM could have a network interface (for example, TAP or TUN) and at the host side you enable IP forwarding, then you configure your routing table so each VM's IP address is routed to the respective VM's TAP/TUN. The default route should go outside to the Internet.

Another solution could be via bridging:
Each VM has its own TAP interface, then you bridge all of them together with the physical interface of the host. You don't need to route, each VM would then have direct access to its own IP packets. This setup has the disadvantage that each VM could sniff all the network traffic of all the VMs and the host, so it's not so tight securitywise.

So, depending on your performance and security requirements there are different options. Bridging would be the simplest to configure, I think.

What's for sure, you don't need a NAT because you have external addresses to map. NAT is used for hiding multiple hosts behind a single external IP address. You just need simple routing.
 
It partly depends how the addresses have been routed to you.

It seems like your main gateway is 46.4.50.1, which your server has an address on. The additional addresses are on a completely different range of 49.12.49.x.

If you just assign one of these addresses to a guest, it won't have a gateway, and if you try to use 46.4.50.1, it won't know how to get to it. (Usually you get an error like you show that it refuses to set it)

I would try assigning one of the addresses to the host, on the virtual switch / bridge seems the best place. Assign one of the remaining addresses to the guest and use the address you gave the host as the gateway. This will allow traffic to leave the guest and hit the virtual switch on the host. The host can then route this to its default gateway. (You'll need to have gateway_enable="yes" on the host.)
 
Not sure if you blanked out the netmask instead of ip by accident in the original post...

Just to add though, even if you did want to assign all the addresses to the host, which you more than likely don't, you should never be assigning the network or broadcast addresses. It's also standard to use a 255.255.255.255 (/32) netmask for all but the first address when more than one address are on the same subnet.
 
It partly depends how the addresses have been routed to you.

It seems like your main gateway is 46.4.50.1, which your server has an address on. The additional addresses are on a completely different range of 49.12.49.x.

If you just assign one of these addresses to a guest, it won't have a gateway, and if you try to use 46.4.50.1, it won't know how to get to it. (Usually you get an error like you show that it refuses to set it)

I would try assigning one of the addresses to the host, on the virtual switch / bridge seems the best place. Assign one of the remaining addresses to the guest and use the address you gave the host as the gateway. This will allow traffic to leave the guest and hit the virtual switch on the host. The host can then route this to its default gateway. (You'll need to have gateway_enable="yes" on the host.)

Thanks that did the trick I assigned an IP to the virtual switch then used it as a gateway IP for the vm.
The vm now has access to the internet with a public IP address.
Thanks for the tip about the netmask I was using /32 for the usable address and I did blank out the netmask by accident instead of ip's lol..oh well
 
Back
Top