bhyve iaas

Z

zader

#1 What a good toolset for iaas on bare metal freebsd?
(already using ansible for management) but I don't see anything that works well or can no touch deploy vm's
I also have some basic scripts to deploy jails, but would like fully convert that to no touch install.

#2 On metal, I want to passthrough an nvida gird c16 8000 gpu..
Is it possible to pass it through now on bhyve?
and if so, is it possible to passthrough 8 cores to one vm and 8 cores to another similar to esxi?


thanks!
 
zader said:
(already using ansible for management) but I don't see anything that works well or can no touch deploy vm's
I also have some basic scripts to deploy jails, but would like fully convert that to no touch install.
Click to expand...
You might want to take a look at devel/libvirt. There should be some Ansible modules floating around that allow you to use Ansible to manage the VMs through libvirt.
 
I think a worthwhile project for someone might be to implement k8s on FreeBSD. Installing and using Docker and Kubernetes on FreeBSD provides a roadmap.

We don't have a docker or podman repo for FreeBSD. A docker repo would be useful in the FreeBSD ecossystem and maybe people might use it. A more useful option may be to *also* adopt the Linux docker/podman repos for linuxulator containers.

Having said that, FreeBSD jails are not the same as Linux containers. Linux containers are built on top of cgroups and namespaces, both of which IMO are not nearly as secure as jails but they may be different enough for podman containers to "notice." The only way we could know for sure would be to try it.

I think the step-by-step plan might be:
  1. Implement a jail management framework, using k8s and docker, that could be immediately used by jails and any future Linux container emulation layer we may wish to develop. (Basically the topic of this thread.)
  2. Create a FreeBSD docker container repo populated with jail images using the FreeBSD ABI.
  3. If Linux containers (cgroups + namespaces) are different enough from our jails, implement a shim layer (hopefully that's all that's needed) to allow jails to emulate Linux containers).
Just some food for thought.
 
zader said:
is it possible to passthrough 8 cores to one vm and 8 cores to another similar to esxi?
Click to expand...
I seriously doubt you could split the video cores of a GPU. I have passed thru GPU on nvidia.
Passthru is acheived by PCI address of the card and pptdevs

vmm(4)

www.freebsd.org www.freebsd.org

So for example a network card with 4 interfaces might have 4 seperate PCI bus addresses.
Allowing you to pass through 4 independant interfaces.
So look at your GPU with pciconf. Does it present different bus addresses?
(Don't include the GPU's HDMI Audio interface.)
Same with USB cards. Some USB cards present all their interfaces to one PCI address while others present seperate PCI addresses for each of the USB ports.

I am afraid you will be constrained by FreeBSD interpretation of your hardware.
I know nothing about ESXi and how it splits video cores so I can't comment.

libvirt: Bhyve driver

libvirt, virtualization, virtualization API
libvirt.org libvirt.org
Cut and paste sample XML. Tailor to suit your images.
Fire up virsh command prompt to start VM.
 
Several methods with CBSD:

1) CBSDfile method ( also for jail; also for API/remote cluster by adding CLOUD_\* vars: requires API/remote cluster or MyBee Appliance )

Notes: the cloud- prefix in profile means that CBSD will use cloud-init here, not ISO ( as suggested by rootbert ^ )

Code: 
#CLOUD_URL="https://us-sw.api.my.domain"
#CLOUD_KEY="ssh-ed25519 XXX your@comment"

# cloud CentOS 7
bhyve_vm1()
{
        vm_ram="1g"
        vm_cpus="1"
        imgsize="6g"
        vm_os_type="linux"
        vm_os_profile="cloud-CentOS-7-x86_64"
        interface="auto"                # can be VALE, e.g: interface="vale1"
        ci_fqdn="${jname}.example.com"

        ci_ip4_addr=$( dhcpd )          # use CBSD 'dhcp' script to get free IP from CBSD pool
                                        # or use static IP, e.g: ci_ip4_addr="192.168.0.50"
        ci_gw4="192.168.0.50"           # IPv4 gateway, can set globally via globals or ~cbsd/etc/bhyve-default-default.conf
        # other bhyve-related settings:
        # bhyve_vnc_tcp_bind="0.0.0.0"
        # bhyve_generate_acpi="1"
        # bhyve_wire_memory="1"

}

# CentOS 9
bhyve_vm2()
{
        vm_ram="1g"
        vm_cpus="1"
        imgsize="6g"
        vm_os_type="linux"
        vm_os_profile="cloud-CentOS-stream-9-x86_64"
        ci_fqdn="${jname}.example.com"

        ci_ip4_addr=$( dhcpd )          # use CBSD 'dhcp' script to get free IP from CBSD pool
                                        # or use static IP, e.g: ci_ip4_addr="192.168.0.50"
        ci_gw4="192.168.0.50"           # IPv4 gateway
}

# Debian 11
bhyve_vm3()
{
        vm_ram="1g"
        vm_cpus="1"
        imgsize="6g"
        vm_os_type="linux"
        vm_os_profile="cloud-Debian-x86-11"
        interface="auto"                # can be VALE, e.g: interface="vale1"
        ci_fqdn="${jname}.example.com"

        ci_ip4_addr=$( dhcpd )          # use CBSD 'dhcp' script to get free IP from CBSD pool
                                        # or use static IP, e.g: ci_ip4_addr="192.168.0.50"
        ci_gw4="192.168.0.50"           # IPv4 gateway
        ci_interface_mtu="1500"         # MTU
 
}

# Ubuntu 20
bhyve_vm4()
{
        vm_ram="1g"
        vm_cpus="1"
        imgsize="6g"
        vm_os_type="linux"
        vm_os_profile="cloud-ubuntuserver-amd64-20.04"
        interface="auto"                # can be VALE, e.g: interface="vale1"
        ci_fqdn="${jname}.example.com"

        ci_ip4_addr=$( dhcpd )          # use CBSD 'dhcp' script to get free IP from CBSD pool
                                        # or use static IP, e.g: ci_ip4_addr="192.168.0.50"
        ci_gw4="192.168.0.50"           # IPv4 gateway
}

# FreeBSD 13 UFS
bhyve_vm5()
{
        vm_ram="1g"
        vm_cpus="1"
        imgsize="6g"
        vm_os_type="freebsd"
        vm_os_profile="cloud-FreeBSD-ufs-x64-13.1"
        interface="auto"                # can be VALE, e.g: interface="vale1"
        ci_fqdn="${jname}.example.com"

        ci_ip4_addr=$( dhcpd )          # use CBSD 'dhcp' script to get free IP from CBSD pool
                                        # or use static IP, e.g: ci_ip4_addr="192.168.0.50"
        ci_gw4="192.168.0.50"           # IPv4 gateway
}

Just run 'cbsd up' or 'cbsd up vm3' and after few seconds you can login into VM via 'cbsd blogin'


2) Puppet module: https://forge.puppet.com/modules/olevole/cbsd ( also for jail )
3) Rex module: https://github.com/VVelox/Rex-Virtualization-CBSD
4) CBSD API + nubectl ( thin client for Windows, MacOS, Linux and BSD): https://www.bsdstore.ru/en/cbsd_api_ssi.html ( requires API/remote cluster or MyBee Appliance; also for jail )
5) from CLI:
Code: 
cbsd bcreate jname=c1 vm_ram=4g vm_cpus=2 vm_os_type=freebsd vm_os_profile=cloud-FreeBSD-ufs-x64-13.0 imgsize=20g ci_ip4_addr=10.0.1.88 ci_gw4=10.0.1.3 runasap=1
( also for jail )
6) via TUI: `cbsd bconstruct-tui` ( also for jail )

PS: Someone wrote about the Terraform provider for CBSD, but I could not find the information - you can probably ask in the CBSD chat.

And finally: CBSD supports for bhyve GPU Passthru ( + GOP, graphics output: see /usr/ports/sysutils/cbsd port options ) out of the box: Windows bhyve + GOP, Linux bhyve + FreeBSD bhyve + GOP ( never tested myself )
 
  • Thanks
Reactions: Ole
Purkuapas said:
PS: Someone wrote about the Terraform provider for CBSD, but I could not find the information - you can probably ask in the CBSD chat.
Click to expand...

Terraform is currently not supported, but work is underway on writing a provider for CBSD. I expect it to be available by the end of the year.
Perhaps a Reggae (by Goran Mekić) can also be added to the list of candidates for bhyve+IaC. But this is also a CBSD satellite.
 
You must log in or register to reply here.
Back
Top