bhyve Current state of bhyve Nvidia passthrough?

glzrr · Aug 29, 2025

i have odd sitation .
freebsd 14.2 - brand new with HETZNER machine .
this is an intel machine :

this is my loader conf :

Code:

hw.vmm.enable_vtd="1"
pptdevs="111/0/0 111/0/1"
vmm_load=”YES”
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"

Code:

vm passthru >>
vgapci1    111/0/0      No           AD102GL [RTX 6000 Ada Generation]
hdac0      111/0/1      No           AD102 High Definition Audio Controller

yet the card seems not to load PPT driver .

pciconf :

Code:

vgapci1@pci0:111:0:0:   class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x26b1 subvendor=0x10de subdevice=0x16a1
    vendor     = 'NVIDIA Corporation'
    device     = 'AD102GL [RTX 6000 Ada Generation]'
    class      = display
    subclass   = VGA
hdac0@pci0:111:0:1:     class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x22ba subvendor=0x10de subdevice=0x16a1
    vendor     = 'NVIDIA Corporation'
    device     = 'AD102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA

any clues ?

SirDice · Aug 29, 2025

Is this the same card that provides the output on the console?

ZioMario · Aug 29, 2025

Try this command and paste the output :

Code:

# devctl set driver -f pci0:111:0:0 ppt

also try this :

Code:

pptdevs="111/0/0"

glzrr · Aug 29, 2025

ZioMario said:
Try this command and paste the output :

Code:

# devctl set driver -f pci0:111:0:0 ppt

also try this :

Code:

pptdevs="111/0/0"

weird the set driver worked no output . vm passthru now states its working .
seend like loader.conf is not working .
the options to use just the first one i already did with no luck .

ClintBSD · Aug 29, 2025

ZioMario said:
Does it work with FreeBSD 14.3 ?

Yes, I also upgraded to 14.3 and replaced the bhyve and vmm.ko files and it works for both windows and linux:

 cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve

cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko

cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko

ZioMario · Aug 30, 2025

glzrr said:
weird the set driver worked no output . vm passthru now states its working .
seend like loader.conf is not working .
the options to use just the first one i already did with no luck .

I didn't not understand what works and what does not work now for u.

glzrr · Aug 30, 2025

ZioMario said:
I didn't not understand what works and what does not work now for u.

sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .

T-Aoki · Aug 31, 2025

glzrr said:
sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .

If running the command after boot helps, isn't putting it in /etc/rc.conf (or if you want to separate it from other configs, /etc/rc.local.conf) work?

It's not a good workaround, but as these configuration files are actually run as parts of /etc/rc "script" to set variables, it would work.

ZioMario · Aug 31, 2025

glzrr said:
sorry ,
on each reboot if i do
devctl set driver -f pci0:111:0:0 ppt - its working. and i can passthrough the card to bhyve .
still i cannot make this survive a boot . hence /boot/loader method is not working for me .

You can create something like this script :

Code:

nano run-bhyve :

#!/usr/local/bin/bash

if ! pciconf -l pci0:111:0:0 | grep -q "^ppt"; then
echo "0:111:0:0 is not attached to ppt,attaching..."
devctl detach pci0:111:0:0
devctl clear driver -f pci0:111:0:0 ppt
devctl set driver -f pci0:111:0:0 ppt
else
echo "0:111:0:0 already attached to ppt"
fi

if ! pciconf -l pci0:111:0:1 | grep -q "^ppt"; then
echo "0:111:0:1 is not attached to ppt,attaching..."
devctl detach pci0:111:0:1
devctl clear driver -f pci0:111:0:1 ppt
devctl set driver -f pci0:111:0:1 ppt
else
echo "0:111:0:1 already attached to ppt"
fi

bhyve -S -c sockets=8,cores=1,threads=1 -m 16G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/zroot-133/bhyve/img/Linux/Ubuntu-2404-KDE6-Wayland.img,bootindex=1 \
-s 8:0,passthru,0:111:0:0 \
-s 8:1,passthru,0:111:0:1 \
-s 11,hda,play=/dev/dsp,rec=/dev/dsp \
-s 13,virtio-net,tap4 \
-s 14,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:4 < /dev/null & sleep 5 && vncviewer 0:4 &

Code:

chmod +x run-bhyve
./run-bhyve

redmog · Aug 31, 2025

glzrr said:
i have odd sitation .
freebsd 14.2 - brand new with HETZNER machine .
this is an intel machine :

this is my loader conf :

Code:

hw.vmm.enable_vtd="1" pptdevs="111/0/0 111/0/1" vmm_load=”YES” kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" cryptodev_load="YES" zfs_load="YES"

Is this a copy-and-paste error? Because the quotes are different in this line:
vmm_load=”YES”

So it is possible that the loader LUA is interpreting this as:
vmm_load="”YES”"

... and as the value does not match YES then the vmm module is not loaded at boot ...?

glzrr · Aug 31, 2025

redmog said:
Is this a copy-and-paste error? Because the quotes are different in this line:
vmm_load=”YES”

So it is possible that the loader LUA is interpreting this as:
vmm_load="”YES”"

... and as the value does not match YES then the vmm module is not loaded at boot ...?

i will try again by hand . thank you

GpuOperator · Sep 2, 2025

I have some questions about Corvin's patch to FreeBSD 14.2: f241d2b
Half of the patch targets 0x88000 and half targets 0x880000. The commit message said it was the latter, but is that right or perhaps half is missing a 0?
There is also a problem in the script on https://dflund.se/~getz/Notes/2024/freebsd-gpu/ if it is taken literally

Code:

cd include
make -j64
make install 
cd lib/libvmmapi
make -j64
make install 
cd sys/modules/vmm
make -j64
make install 
cd usr.sbin/bhyve
make -j64
make install 
cd usr.sbin/bhyvectl
make -j64
make install 
cd usr.sbin/bhyveload
make -j64
make install

The cd's are all relative to the root directory of the source tree but there is no cd back to it after each make and make install.
The net result of these makes me wonder if the patch is needed at all to get GPU passthru to work on bhyve. My experience is that nvidia-smi works on FreeBSD 14.2 without any patch, though I have not any luck with CUDA and the patch did not help. Could anyone confirm whether the patch actually makes a difference?

ClintBSD · Sep 14, 2025

GpuOperator said:
I have some questions about Corvin's patch to FreeBSD 14.2: f241d2b
Half of the patch targets 0x88000 and half targets 0x880000. The commit message said it was the latter, but is that right or perhaps half is missing a 0?
There is also a problem in the script on https://dflund.se/~getz/Notes/2024/freebsd-gpu/ if it is taken literally

Code:

cd include make -j64 make install cd lib/libvmmapi make -j64 make install cd sys/modules/vmm make -j64 make install cd usr.sbin/bhyve make -j64 make install cd usr.sbin/bhyvectl make -j64 make install cd usr.sbin/bhyveload make -j64 make install

The cd's are all relative to the root directory of the source tree but there is no cd back to it after each make and make install.
The net result of these makes me wonder if the patch is needed at all to get GPU passthru to work on bhyve. My experience is that nvidia-smi works on FreeBSD 14.2 without any patch, though I have not any luck with CUDA and the patch did not help. Could anyone confirm whether the patch actually makes a difference?

Refer to my post #128 on page 6. Bhyve works in a windows VM with either string:

C:

static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

or

C:

static const char bhyve_id[12] = "Bhyve Bhyve ";

A linux virtual machine works with the string below:

C:

static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

Once you have compiled the code, if you upgrade your system, you can simply replace the binaries below with your patched versions:

 cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve

cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko

cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko

c-c-c-c · Sep 14, 2025

ClintBSD said:
Refer to my post #128 on page 6. Bhyve works in a windows VM with either string:

C:

static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

or

C:

static const char bhyve_id[12] = "Bhyve Bhyve ";

A linux virtual machine works with the string below:

C:

static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

Once you have compiled the code, if you upgrade your system, you can simply replace the binaries below with your patched versions:
cp -i ~/bhyve/bhyve_lin /usr/sbin/bhyve cp -i ~/bhyve/vmm_lin.ko /boot/modules/vmm.ko cp -i ~/bhyve/vmm_lin.ko /boot/kernel/vmm.ko

please learn how to use patch(1)

ZioMario · Sep 14, 2025

I think that the time is came for me for going to the nvidia forum,asking for a custom bhyve_id,tailored only for bhyve. Bhyve should be not confused anymore with kvm. They don't care so much for linux and FreeBSD,so we should be lucky.

zldrobit · Sep 17, 2025

A new GPU passthrough patch D51892 was just committed on the 16-CURRENT (main branch) by Corvink. Maybe someone could build the source and test it.

ClintBSD · Sep 24, 2025

c-c-c-c said:
please learn how to use patch(1)

I did use the patch file and created my own patch files. I just wanted to be clear how I got this to work. I used your original patch file for 14.0.

zldrobit · Sep 24, 2025

zldrobit said:
A new GPU passthrough patch D51892 was just committed on the 16-CURRENT (main branch) by Corvink. Maybe someone could build the source and test it.

This patch was cherry-picked in stable/15 and stable/14, and will probably be in 15-ALPHA4 but not in 15-ALPHA3.

daniel_jimenez · Sep 27, 2025

Is there a recommended procedure for building a patched bhyve for nvidia gpu passthrough on 14.3 at the moment?
My target guest OSs are mx linux, windows, and freebsd.

I'll try the patch file by ClintBSD, but I'm confused whether it's the most current or if it includes the latest stuff, and also if it will work as is on both windows and linux.

Will this be incorporated into 14.4 and/or 15.0 by the time they hit release?

It seems like there's a lot of people who have been testing and reporting here, perhaps a webpoll could help identify if there are still outstanding issues and figure out solutions.

T-Aoki · Sep 28, 2025

daniel_jimenez said:
Will this be incorporated into 14.4 and/or 15.0 by the time they hit release?

Should be.

Both stable/14 and stable/15 has it MFC'ed. So next release from both should have it. Not sure any plan exists for further MFS (Merge From Stable) to releng/14.3 or not. And releng/14.3 doesn't seem to have it at least for now.

daniel_jimenez · Sep 28, 2025

Here's the steps I followed, hopefully it helps someone:

cd /usr/src
fetch https://forums.freebsd.org/attachments/build-txt.21848/ -o ./build.sh
chmod 700 ./build.sh
git clone https://github.com/freebsd/freebsd-src.git freebsd
fetch https://forums.freebsd.org/attachments/freebsd-14-2-beckhoff-linux-patch-txt.23368/ -o freebsd-14-2-beckhoff-linux.patch.txt
cd freebsd
git fetch origin releng/14.3
git checkout releng/14.3
patch -p1 < ../freebsd-14-2-beckhoff-linux.patch.txt
cd ../
./build.sh --without-bhf --verbose --src-dir=/usr/src/freebsd
reboot

After reboot, it works!

This is using a quadro p4000 gpu.

I've only tested mx linux, haven't tested shutting down the vm and turning it back on.
Haven't tested windows, either.

Important question, is there an additional benefit to the D51892 patch?

Thanks to all involved people, especially Corvin, ClintBSD, an ZioMario.

Daniel

bhyve Current state of bhyve Nvidia passthrough?

Administrator