bhyve vm-bhyve Ubuntu 24.04 installed

NapoleonWils0n

NapoleonWils0n

Ubuntu 24.04 installed with vm-bhyve after a bit of work

github.com

cerberus/freebsd/vm-bhyve.org at master · NapoleonWils0n/cerberus

cerberus code library. Contribute to NapoleonWils0n/cerberus development by creating an account on GitHub.
github.com github.com

install packages

Code: 
doas pkg install bhyve-firmware grub2-bhyve edk2-bhyve vm-bhyve-devel

create the zfs datasets

Code: 
doas zfs create -o mountpoint=/vm zroot/vm
doas zfs create -o mountpoint=/vm/.iso zroot/vm/iso
doas zfs create -o mountpoint=/vm/.templates zroot/vm/templates

rc.conf

Code: 
doas sysrc kld_list+="vmm nmdm"
doas sysrc vm_enable=YES
doas sysrc vm_dir="zfs:zroot/vm"

kldload

only needed if you havent rebooted, and not needed after that

Code: 
doas kldload vmm nmdm

networking,
pinched the tips from vermaden

vermaden.wordpress.com

FreeBSD Bhyve Virtualization

The Bhyve FreeBSD hypervisor (called/spelled ‘beehive’ usually) was created almost 10 years ago. Right now it offers speed and features that other similar solutions provide – such…
vermaden.wordpress.com vermaden.wordpress.com

gateway enable

Code: 
doas sysrc gateway_enable=YES

sysctl

Code: 
doas sysctl net.link.tap.up_on_open=1
doas sysctl net.inet.ip.forwarding=1

set options permanently

Code: 
doas echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
doas echo "net.inet.ip.forwarding=1" >> /etc/sysctl.conf

copy templates

Code: 
doas cp -a /usr/local/share/examples/vm-bhyve /vm/.templates

doas set up

edit doas.conf and add a rule for the vm command

Code: 
doas vi /usr/local/etc/doas.conf

change username to your username

Code: 
# vm-bhyve
permit nopass :username cmd vm

start the vm service
only needed if you havent rebooted, and not needed after that


Code: 
doas service vm start

vm init

Code: 
doas vm init

networking

change lagg0 to your network interface

Code: 
doas vm switch create public
doas vm switch add public lagg0

fetch the iso

i found the vm iso command stopped downloading the iso part way through,
so instead i used fetch to download the iso

Code: 
doas fetch -o /vm/.iso/ubuntu-24.04.1-live-server-amd64.iso 'https://releases.ubuntu.com/24.04.1/ubuntu-24.04.1-live-server-amd64.iso'

find the number of cpu's on your machine

Code: 
sysctl kern.sched.topology_spec

edit the ubuntu template

Code: 
doas vi /vm/.templates/ubuntu.conf

i have set the cpu to 6 as i have 12 cpus
and the memory to 8G because i have 16G

im need a bit of power for the apps im going to be running

Code: 
loader="grub"
cpu=6
memory=8G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
grub_run_partition="2"
grub_run_dir="/grub"

this is the important part of the config

Code: 
grub_run_partition="2"
grub_run_dir="/grub"

if you dont have those two option you will be dumped into the grub prompt
and have to load some stuff manually which didnt work for me

create the vm

Code: 
doas vm create -t ubuntu -s 40G ubuntu

-t is the template name
-s is the size
ubuntu is the name of the vm

vm install

use the -f option to run the vm in the current terminal

Code: 
doas vm install -f ubuntu ubuntu-24.04.1-live-server-amd64.iso

just got everything installed
so i thought id share my notes so far
 
Thank you for sharing this.

NapoleonWils0n said:
if you dont have those two option you will be dumped into the grub prompt
and have to load some stuff manually which didnt work for me
Click to expand...
You should be able to prevent this issue just by using UEFI instead:
github.com

Ubuntu 24.04 guest requires the grub run partition setting · Issue #550 · churchers/vm-bhyve

Starting with the Ubuntu 24.04 server (amd64 ISO) the installation completes successfully. This is using the suggested "Debian and Ubuntu (Grub)" section on the Wiki page However when rebooting, th...
github.com github.com
 
cheers mate
ill try using uefi instead

still good to work out what the issue with grub is
im not going to be using graphics so grub will do for now

i want to pass through my gpu
and install the nvidia driver and cuda

however i only have 1 gpu

i know if you have 2 gpus you can "blacklist" one of them
and then pass it though to the vm

just wondering how to go about passing through the gpu
without "blacklisting" it on Freebsd

i guess you can use the vm passthru option

on the nvidia side on things

with jails you have to manually install the exact version of the nvidia driver used on Freebsd for it to work

just wondering if anyone has passed through a single gpu
and what the general set up is

i was hoping to install janus-pro

github.com

GitHub - deepseek-ai/Janus: Janus-Series: Unified Multimodal Understanding and Generation Models

Janus-Series: Unified Multimodal Understanding and Generation Models - deepseek-ai/Janus
github.com github.com

and whisperx

github.com

GitHub - m-bain/whisperX: WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)

WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization) - m-bain/whisperX
github.com github.com

googling for information these days is awful
so if anyone can give me a kick in the right direction that would be great
 
vm passthru

Code: 
doas vm passthru

Code: 
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           8th Gen Core Processor Host Bridge/DRAM Registers
pcib1      0/1/0        No           6th-10th Gen Core Processor PCIe Controller (x16)
vgapci1    0/2/0        No           CoffeeLake-H GT2 [UHD Graphics 630]
none0      0/4/0        No           Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem
none1      0/8/0        No           Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
pchtherm0  0/18/0       No           Cannon Lake PCH Thermal Controller
xhci0      0/20/0       No           Cannon Lake PCH USB 3.1 xHCI Host Controller
none2      0/20/2       No           Cannon Lake PCH Shared SRAM
ig4iic0    0/21/0       No           Cannon Lake PCH Serial IO I2C Controller
ig4iic1    0/21/1       No           Cannon Lake PCH Serial IO I2C Controller
none3      0/22/0       No           Cannon Lake PCH HECI Controller
ahci0      0/23/0       No           Cannon Lake Mobile PCH SATA AHCI Controller
pcib2      0/27/0       No           Cannon Lake PCH PCI Express Root Port
pcib7      0/28/0       No           Cannon Lake PCH PCI Express Root Port
pcib8      0/28/4       No           Cannon Lake PCH PCI Express Root Port
pcib9      0/29/0       No           Cannon Lake PCH PCI Express Root Port
isab0      0/31/0       No           Cannon Lake LPC Controller
hdac0      0/31/3       No           Cannon Lake PCH cAVS
ichsmb0    0/31/4       No           Cannon Lake PCH SMBus Controller
none4      0/31/5       No           Cannon Lake PCH SPI Controller
vgapci0    1/0/0        No           TU117M [GeForce GTX 1650 Mobile / Max-Q]
pcib3      2/0/0        No           JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
pcib4      3/0/0        No           JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
pcib5      3/1/0        No           JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
pcib6      3/2/0        No           JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
none5      4/0/0        No           JHL6340 Thunderbolt 3 NHI (C step) [Alpine Ridge 2C 2016]
xhci1      58/0/0       No           JHL6340 Thunderbolt 3 USB 3.1 Controller (C step) [Alpine Ridge 2C 2016]
iwlwifi0   59/0/0       No           Wi-Fi 6 AX200
rtsx0      60/0/0       No           RTS525A PCI Express Card Reader
nvme0      61/0/0       No           2300 NVMe SSD [Santana]

Code: 
nvidia-smi

Code: 
Wed Jan 29 15:34:16 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1650        Off |   00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8              2W /   50W |       1MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

so i guess this is the gpu

Code: 
vgapci0    1/0/0        No           TU117M [GeForce GTX 1650 Mobile / Max-Q]
 
the READY column lists devices that are ready to be passed through
and the gpu is listed as No because i havent blacklisted it

im sure someone on the forum mention its possible to use the gpu without blacklisting it
but my memory may be getting a bit fuzzy

just watching this video to see if it has any tips

 
no joy

ubuntu.conf

Code: 
loader="grub"
cpu=6
memory=8G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
grub_run_partition="2"
grub_run_dir="/grub"

tried both these options
1/0/0 is the gpu on the freebsd host

1 - passthru0

Code: 
passthru0="1/0/0=7:0"

2 - bhyve_option

Code: 
bhyve_options="-s 2:0,passthru,1/0/0"

the vm wouldnt boot

i cant mask the gpu because i only have 1 gpu on the machine

monwarez mentioned the following
have to read up on that

Just need to implement the virtio-gpu with bhyve as host, using virglrenderer with the drm-renderers option.
Click to expand...

monwarez

Post in thread 'Ollama working with deepseek-r1, deepseek-coder, mistral, Nvidia gpu and emacs with gptel on Freebsd 14.2'

NapoleonWils0n said:
thinking out load

i wonder if torch will work in a linux byhve vm with my gpu

however i only have one gpu,
so i cant disable the gpu to pass through to the byhve vm
Click to expand...
Just need to implement the virtio-gpu with bhyve as host, using virglrenderer with the drm-renderers option.
  • Thanks
 
podman works

Code: 
sudo apt install podman podman-compose

Code: 
podman run quay.io/podman/hello

podman.png
 
Oh, im all day today fighting with virtualization but im using vermandens how to and trying arch linux ... for some reasons i have loads of issues and cant start . tried virt-manager - errors, etc. let me try your one. i hope i have my patched kernel working for gpu passtrough so we will c where i land.
P.s. im not really a big fan of ubuntu as a lot of people have issues using ubuntu for local llm`s ill try to do with debian.
 
ignore the following post
i missed a step in the set up

vm-bhyve and ubuntu have stopped working for me

i just get dumped to a grub prompt

Code: 
Connected
grub> ls
(hd0) (host)

which doesnt list as many devices as it did before
nothing in (hd0) or (host), so i cant manually load anything

ubuntu.conf for vm-bhyve

using grub_run_partition="2" in the ubuntu.conf
stops you being dumped to the grub prompt and gets you to the login screen

but its not working anymore,
and it still dumps me to the grub prompt with it commented out

Code: 
loader="grub"
cpu=6
memory=8G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
grub_run_partition="2"
grub_run_dir="/grub"

my existing ubuntu vm still works

but if i create a new ubuntu vm im just dumped to the grun prompt
and cant manually load anything to boot

i upgraded to 14 p1 yesterday
nothing in any of my config has changed

rc.conf and vm templates are still the same

so not sure what the issue is

doh my bad
i missed a step creating the vm

forgot to attach the iso

Code: 
doas vm install -f ubuntu ubuntu-24.04.1-live-server-amd64.iso
 
Is it necessary for you to use grub?
Im running a few Ubuntu VMs with the following.

Code: 
loader="uefi"
cpu=1
memory=4096M
network0_type="virtio-net"
network0_switch="vm_bridge0"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
graphics="yes"
xhci_mouse="yes"
graphics_res="1024x768"
 
Does this qualifies gpu passtrough now and i can pass it to vm?
Its shown under command vm passthru
Code: 
ppt0       129/0/0      Yes          GV100GL [Quadro GV100]
ppt1       129/0/1      Yes          -
 
victort said:
Is it necessary for you to use grub?
Im running a few Ubuntu VMs with the following.

Code: 
loader="uefi"
cpu=1
memory=4096M
network0_type="virtio-net"
network0_switch="vm_bridge0"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
graphics="yes"
xhci_mouse="yes"
graphics_res="1024x768"
Click to expand...

im running invidious in the bhyve vm
which doesnt need graphics

so i just used grub
 
GlitchyDot said:
Does this qualifies gpu passtrough now and i can pass it to vm?
Its shown under command vm passthru
Code: 
ppt0       129/0/0      Yes          GV100GL [Quadro GV100]
ppt1       129/0/1      Yes          -
Click to expand...

i think you can do this in your bhyve template to passthru that device

Code: 
passthru0="129/0/0 129/0/1"


or this

Code: 
bhyve_options="-s 08:0,passthru,129/0/0 -s 09:0,passthru,129/0/1"

08:00 and 09:00 are the pci slots on the vm
 
to get the networking i only had to run this

replace lagg0 with your network interface

Code: 
sudo vm switch create public
sudo vm switch add public lagg0
 
changed to uefi:
Code: 
BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0): Not Found

>>Start PXE over IPv4.
 
Success , but partial. no internet and dont know if cuda works due to no internet.
but inside debian it shows me my ip but i oculd not install with apt mirror and ping does fails.

but vm switch list shows:
Code: 
NAME    TYPE      IFACE      ADDRESS           PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  192.168.1.100/24  no       -    -     igb1
ip a on debian vm - 192.168.1.96/24
 

Attachments

  • nvidia.png
    nvidia.png
    14.4 KB · Views: 237
had to delete my vm and re-create config file from scratch.
grub - no go for me.
uefi - need to add extra line before creating new vm.
so this is how my config looks like:
Code: 
loader="uefi"
cpu=6
memory=4096M
ahci_device_limit="8"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
graphics="yes"
xhci_mouse="yes"
passthru0="129/0/0=2:0"
uefi_vars="yes"
uefi_vars="yes" <--
Maybe i need to change network to e1000 ...
 
Sorted my internet. i dont know difference between virtio and e1000, but i forgot to change back to virt, but my issue was MY MISTAKE. i have 2x lan ports and when i was creating it i used wrong one.
 
I'm going to spam my own page https://srobb.net/vm-bhyve.html. I skimmed the thread, so forgive me if I say something you already mentioned.

To get a GUI you probably need to use uefi as the loader. I've been using, with success, the -t centos7.conf for all my GUI Linuxes. Now, sometimes it won't boot--I sort of cover this on my page but also link to a better explanation at https://www.davidschlachter.com/misc/freebsd-bhyve-uefi-shell.

I do see that vermaden's guides are mentioned in the thread. He knows far more about vm-bhyve than I do.
 
You must log in or register to reply here.
Back
Top