bhyve Current state of bhyve Nvidia passthrough?

i have odd sitation .
freebsd 14.2 - brand new with HETZNER machine .
this is an intel machine :

this is my loader conf :
Code: 
hw.vmm.enable_vtd="1"
pptdevs="111/0/0 111/0/1"
vmm_load=”YES”
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
Code: 
vm passthru >>
vgapci1    111/0/0      No           AD102GL [RTX 6000 Ada Generation]
hdac0      111/0/1      No           AD102 High Definition Audio Controller
yet the card seems not to load PPT driver .

pciconf :
Code: 
vgapci1@pci0:111:0:0:   class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x26b1 subvendor=0x10de subdevice=0x16a1
    vendor     = 'NVIDIA Corporation'
    device     = 'AD102GL [RTX 6000 Ada Generation]'
    class      = display
    subclass   = VGA
hdac0@pci0:111:0:1:     class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x22ba subvendor=0x10de subdevice=0x16a1
    vendor     = 'NVIDIA Corporation'
    device     = 'AD102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
any clues ?
 
Try this command and paste the output :

Code: 
# devctl set driver -f pci0:111:0:0 ppt

also try this :

Code: 
pptdevs="111/0/0"
 
ZioMario said:
Try this command and paste the output :

Code: 
# devctl set driver -f pci0:111:0:0 ppt

also try this :

Code: 
pptdevs="111/0/0"
Click to expand...
weird the set driver worked no output . vm passthru now states its working .
seend like loader.conf is not working .
the options to use just the first one i already did with no luck .
 
ZioMario said:
Does it work with FreeBSD 14.3 ?
Click to expand...
Yes, I also upgraded to 14.3 and replaced the bhyve and vmm.ko files and it works for both windows and linux:

cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve
cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko
cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko
 
glzrr said:
weird the set driver worked no output . vm passthru now states its working .
seend like loader.conf is not working .
the options to use just the first one i already did with no luck .
Click to expand...

I didn't not understand what works and what does not work now for u.
 
ZioMario said:
I didn't not understand what works and what does not work now for u.
Click to expand...
sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .
 
glzrr said:
sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .
Click to expand...
If running the command after boot helps, isn't putting it in /etc/rc.conf (or if you want to separate it from other configs, /etc/rc.local.conf) work?

It's not a good workaround, but as these configuration files are actually run as parts of /etc/rc "script" to set variables, it would work.
 
glzrr said:
sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .
Click to expand...

You can create something like this script :

Code: 
nano run-bhyve :

#!/usr/local/bin/bash

if ! pciconf -l pci0:111:0:0 | grep -q "^ppt"; then
echo "0:111:0:0 is not attached to ppt,attaching..."
devctl detach pci0:111:0:0
devctl clear driver -f pci0:111:0:0 ppt
devctl set driver -f pci0:111:0:0 ppt
else
echo "0:111:0:0 already attached to ppt"
fi

if ! pciconf -l pci0:111:0:1 | grep -q "^ppt"; then
echo "0:111:0:1 is not attached to ppt,attaching..."
devctl detach pci0:111:0:1
devctl clear driver -f pci0:111:0:1 ppt
devctl set driver -f pci0:111:0:1 ppt
else
echo "0:111:0:1 already attached to ppt"
fi

bhyve -S -c sockets=8,cores=1,threads=1 -m 16G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/zroot-133/bhyve/img/Linux/Ubuntu-2404-KDE6-Wayland.img,bootindex=1 \
-s 8:0,passthru,0:111:0:0 \
-s 8:1,passthru,0:111:0:1 \
-s 11,hda,play=/dev/dsp,rec=/dev/dsp \
-s 13,virtio-net,tap4 \
-s 14,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:4 < /dev/null & sleep 5 && vncviewer 0:4 &

Code: 
chmod +x run-bhyve
./run-bhyve
 
glzrr said:
i have odd sitation .
freebsd 14.2 - brand new with HETZNER machine .
this is an intel machine :

this is my loader conf :
Code: 
hw.vmm.enable_vtd="1"
pptdevs="111/0/0 111/0/1"
vmm_load=”YES”
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
Click to expand...

Is this a copy-and-paste error? Because the quotes are different in this line:
vmm_load=”YES”

So it is possible that the loader LUA is interpreting this as:
vmm_load="”YES”"

... and as the value does not match YES then the vmm module is not loaded at boot ...?
 
redmog said:
Is this a copy-and-paste error? Because the quotes are different in this line:
vmm_load=”YES”

So it is possible that the loader LUA is interpreting this as:
vmm_load="”YES”"

... and as the value does not match YES then the vmm module is not loaded at boot ...?
Click to expand...
i will try again by hand . thank you
 
I have some questions about Corvin's patch to FreeBSD 14.2: f241d2b
Half of the patch targets 0x88000 and half targets 0x880000. The commit message said it was the latter, but is that right or perhaps half is missing a 0?
There is also a problem in the script on https://dflund.se/~getz/Notes/2024/freebsd-gpu/ if it is taken literally
Code: 
cd include
make -j64
make install 
cd lib/libvmmapi
make -j64
make install 
cd sys/modules/vmm
make -j64
make install 
cd usr.sbin/bhyve
make -j64
make install 
cd usr.sbin/bhyvectl
make -j64
make install 
cd usr.sbin/bhyveload
make -j64
make install
The cd's are all relative to the root directory of the source tree but there is no cd back to it after each make and make install.
The net result of these makes me wonder if the patch is needed at all to get GPU passthru to work on bhyve. My experience is that nvidia-smi works on FreeBSD 14.2 without any patch, though I have not any luck with CUDA and the patch did not help. Could anyone confirm whether the patch actually makes a difference?
 
GpuOperator said:
I have some questions about Corvin's patch to FreeBSD 14.2: f241d2b
Half of the patch targets 0x88000 and half targets 0x880000. The commit message said it was the latter, but is that right or perhaps half is missing a 0?
There is also a problem in the script on https://dflund.se/~getz/Notes/2024/freebsd-gpu/ if it is taken literally
Code: 
cd include
make -j64
make install
cd lib/libvmmapi
make -j64
make install
cd sys/modules/vmm
make -j64
make install
cd usr.sbin/bhyve
make -j64
make install
cd usr.sbin/bhyvectl
make -j64
make install
cd usr.sbin/bhyveload
make -j64
make install
The cd's are all relative to the root directory of the source tree but there is no cd back to it after each make and make install.
The net result of these makes me wonder if the patch is needed at all to get GPU passthru to work on bhyve. My experience is that nvidia-smi works on FreeBSD 14.2 without any patch, though I have not any luck with CUDA and the patch did not help. Could anyone confirm whether the patch actually makes a difference?
Click to expand...
Refer to my post #128 on page 6. Bhyve works in a windows VM with either string:
C: 
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";
or
C: 
static const char bhyve_id[12] = "Bhyve Bhyve ";

A linux virtual machine works with the string below:
C: 
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

Once you have compiled the code, if you upgrade your system, you can simply replace the binaries below with your patched versions:
cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve
cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko
cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko
 
ClintBSD said:
Refer to my post #128 on page 6. Bhyve works in a windows VM with either string:
C: 
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";
or
C: 
static const char bhyve_id[12] = "Bhyve Bhyve ";

A linux virtual machine works with the string below:
C: 
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

Once you have compiled the code, if you upgrade your system, you can simply replace the binaries below with your patched versions:
cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve
cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko
cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko
Click to expand...
please learn how to use patch(1)
 
I think that the time is came for me for going to the nvidia forum,asking for a custom bhyve_id,tailored only for bhyve. Bhyve should be not confused anymore with kvm. They don't care so much for linux and FreeBSD,so we should be lucky.
 
Is there a recommended procedure for building a patched bhyve for nvidia gpu passthrough on 14.3 at the moment?
My target guest OSs are mx linux, windows, and freebsd.

I'll try the patch file by ClintBSD, but I'm confused whether it's the most current or if it includes the latest stuff, and also if it will work as is on both windows and linux.

Will this be incorporated into 14.4 and/or 15.0 by the time they hit release?

It seems like there's a lot of people who have been testing and reporting here, perhaps a webpoll could help identify if there are still outstanding issues and figure out solutions.
 
Here's the steps I followed, hopefully it helps someone:

cd /usr/src
fetch https://forums.freebsd.org/attachments/build-txt.21848/ -o ./build.sh
chmod 700 ./build.sh
git clone https://github.com/freebsd/freebsd-src.git freebsd
fetch https://forums.freebsd.org/attachments/freebsd-14-2-beckhoff-linux-patch-txt.23368/ -o freebsd-14-2-beckhoff-linux.patch.txt
cd freebsd
git fetch origin releng/14.3
git checkout releng/14.3
patch -p1 < ../freebsd-14-2-beckhoff-linux.patch.txt
cd ../
./build.sh --without-bhf --verbose --src-dir=/usr/src/freebsd
reboot
Click to expand...

After reboot, it works!

This is using a quadro p4000 gpu.

I've only tested mx linux, haven't tested shutting down the vm and turning it back on.
Haven't tested windows, either.

Important question, is there an additional benefit to the D51892 patch?

Thanks to all involved people, especially Corvin, ClintBSD, an ZioMario.

Daniel
 
BuZZok said:
No work on Windows 10 =(
Click to expand...
I'm not using Bhyve myself, but the committed diff doesn't seem to have anything for fingerprints (bhyve bhyve or something else), so if Windows10 assumes anything others only, it shouldn't work without another patch.
 
I'm about to move a server over from Proxmox to FreeBSD. I need to passthrough a Quadro P400 card to a bhyve Debian vm. I'm ever so slightly confused on the patches discussed in this thread. Is D51892 by Corvin enough (as bendany seemed to indicate in #127) or do I still need to patch for the "KVMKVMKVM\0\0\0" string in addition? Because if the merged patch is enough, I'm tempted to install 15.0 Beta instead of 14.3.
 
You must log in or register to reply here.
Back
Top