bhyve Current state of bhyve Nvidia passthrough?

I have two nvidia cards, 1080ti and tesla p100. They worked somehow on 13.2, I was able to passthru these cards to VM with ubuntu and nvidia drivers. In my case the system with 1080ti and p100 on 13.2 is not stable, sometimes happens that whole system freez, or cpu soft lockup occures for infinite time inside VM but it works.

After upgrading to 14.0 (using exactly the same VM, the same drivers, the same bhyve configuration) on VM dmesg shows:
[ 77.208984] NVRM: Can't find an IRQ for your NVIDIA card!
[ 77.212697] NVRM: Please check your BIOS settings.
[ 77.212699] NVRM: [Plug & Play OS] should be set to NO
[ 77.212700] NVRM: [Assign IRQ to VGA] should be set to YES
[ 77.212702] nvidia: probe of 0000:00:07.0 failed with error -1
[ 77.212733] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 77.212734] NVRM: None of the NVIDIA devices were initialized.
[ 77.213269] nvidia-nvlink: Unregistered Nvlink Core, major device number 237
Click to expand...
and repeating system messages:
[ 206.437547] NVRM: Can't find an IRQ for your NVIDIA card!
[ 207.439361] NVRM: Can't find an IRQ for your NVIDIA card!
[ 208.455424] NVRM: Can't find an IRQ for your NVIDIA card!
[ 209.451508] NVRM: Can't find an IRQ for your NVIDIA card!
[ 210.464152] NVRM: Can't find an IRQ for your NVIDIA card!
[ 211.475571] NVRM: Can't find an IRQ for your NVIDIA card!
Click to expand...
I can reproduce this on two different computers (both based on Intel) and two different Nvidia cards (1080ti, p100).
I am looking for any hint.
 
I've been successfully utilizing Corvin's Nvidia passthrough code since version 13.0, specifically with my Quadro K2200 card and Linux (Debian) as the guest OS. Remarkably, I haven't encountered any issues yet. Detailed steps were previously shared by someone in the forum. Essentially, the process involves fetching the codes from https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip for FreeBSD 14.0 compatibility. If targeting version 13.1, simply locate the 13.1/nvidia-wip branch. Then, compile the source code following the instructions outlined in the FreeBSD manual. Finally, proceed with standard passthrough steps as outlined in the bhyve manual. Following these steps should result in a functional Nvidia GPU setup!
 
The code for 14.0 with Corvin patches seems a little bit broken. I needed to do some minor fixes to compile it:
Diff: 
diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index 379b1255a79f..f74adfa2f84b 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -1171,7 +1171,7 @@ passthru_write(struct pci_devinst *pi, int baridx, uint64_t offset, int size,
        } else if (baridx == 0 &&
            pci_get_cfgdata16(pi, PCIR_VENDOR) == PCI_VENDOR_NVIDIA &&
            pci_get_cfgdata8(pi, PCIR_CLASS) == PCIC_DISPLAY) {
-               passthru_cfgwrite(ctx, vcpu, pi, offset - 0x88000, size, value);
+               passthru_cfgwrite(pi, offset - 0x88000, size, value);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1206,7 +1206,7 @@ passthru_read(struct pci_devinst *pi, int baridx, uint64_t offset, int size)
                /* dummy read to MMIO because hw might depend on it */
                memcpy(&val, nvidia_bar0 + offset, size);
 
-               passthru_cfgread(ctx, vcpu, pi, offset - 0x88000, size, (uint32_t *)&val);
+               passthru_cfgread(pi, offset - 0x88000, size, (uint32_t *)&val);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1299,14 +1299,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
@@ -1324,14 +1324,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
(END)
I installed the new kernel and bhyve (only this one file, not the whole world). Now nvidia 1080ti backed to life on 14.0. My guest is ubuntu 22.04 with driver 535.129.03.
I really hope it will be more stable than on 13.2 (without patches) but it needs to do more long term tests.
 
Thank you to everyone involved in developing and testing this work. It is very much needed and appreciated.
My question is: does anyone have a single patch that can be applied to 14.0/STABLE?
I've been a bit disconnected therefore am wondering why (if something of this magnitude exists) it never made it into FreeBSD 14? Thank you.
 
As far as I know there were more patches, and most of them have been applied to 14.0.
For now there are two patches indeed. One is for bhyve (it refers to "dummy bar 0" issue) and the second one is a kernel patch that changes hypervisor signature from bhyve to kvm signature.

I do not know much about "bar 0" and gpu, but for obvious reason you cannot change hypervisor signature permanently. There is a suspicious that nvidia drivers can work differently for different hypervisor.

I tested 1080ti a little more, and it seems to work stable. There is only a problem after restarting VM. That means if I want to restart VM with gpu attached to it, I need to restart the hole computer otherwise the gpu not work.
 

Attachments

Thanks for the patch - but you can easily see why it didn't make it into 14.0. With it applied although Linux passthrough now works, attempting another VM start afterwards almost always results in a hard lock of the entire system at least here (Ivy Bridge, C602 chipset).

The hypervisor signature change isn't required, only the modifications in the bhyve directory. I don't remember it being this unstable in some pre 14.0 builds. It's also currently only Linux that's affected, without the patch Windows and FreeBSD VMs are quite happy to switch between each other, at least in the case of an Nvidia CPU with proprietary (Windows) and open source (FreeBSD) drivers on my system.

Unfortunately Linux passthrough is what I need, because I'm trying to compare FreeBSD WINE to Linux WINE, as I suspect Linux WINE is more compatible than FreeBSD (even for the same version).
 
I got Ubuntu 24.04 working on FreeBSD 14.2 via Bhyve and I'm amazed how it consumes less 1% CPU resources while idling.
Anyone knows if "RTX 3080 Ti" GPU is supported for Bhyve GPU Pass through, where guest is Ubuntu?

Thanks.
 
**That's how this workaround should look like:**

At this moment I do not think 15-CURRENT will work because Corvin did some updates and it is totally not working unless he fixed it.

A and B steps were enough to run Windows 10 with an Nvidia GPU for me. 11 hangs with more than one CPU. If you later only need to update the system, it may be enough to run A, B, and G.

A.1) If you do not have the source, you can clone:
Code: 
git clone -b releng/14.1 https://git.freebsd.org/src.git /usr/src
cd /usr/src

B.1) Next, edit the file sys/amd64/vmm/x86.c and find the following line:
Code: 
static const char bhyve_id[12] = "Bhyve Bhyve ";

B.2) Change it to:
Code: 
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

B.3) Then, build and install the kernel:
Code: 
make -j$(nproc) buildkernel
make -j$(nproc) installkernel

C.1) Now let’s build world:
Code: 
make -j$(nproc) buildworld

D.1) Get Corvin’s patches:
Code: 
git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src /usr/src-nvidia

E.1) We need to change the buildworld object directory:
Code: 
export MAKEOBJDIRPREFIX=/usr/obj-nvidia

E.2) Now we need to install includes and make the rest:
Code: 
cd /usr/src-nvidia
cd include
make -j$(nproc)
make install 
cd -
cd lib/libvmmapi
make -j$(nproc)
cd - 
cd sys/modules/vmm
make -j$(nproc)
cd - 
cd usr.sbin/bhyve
make -j$(nproc)
cd - 
cd usr.sbin/bhyvectl
make -j$(nproc)
cd - 
cd usr.sbin/bhyveload
make -j$(nproc)
cd -

F.1) For step F, you need to reboot the machine and in Boot Loader select the second variant 'Single User'. After the kernel boots, mount the disks:

For UFS:
Code: 
mount -a

For ZFS:
Code: 
zfs set readonly=off zroot
zfs mount -a

F.2) Let’s install World:
Code: 
cd /usr/src
etcupdate -p
make -j$(nproc) installworld
etcupdate -B

F.3) Delete old stuff that's not needed anymore:
Code: 
make delete-old
make delete-old-libs

Or if you know what you are doing:
Code: 
yes | make delete-old
yes | make delete-old-libs

G.1) At the end, let’s install Corvin’s patched Bhyve:
Code: 
cd /usr/src-nvidia
export MAKEOBJDIRPREFIX=/usr/obj-nvidia
cd lib/libvmmapi
make install
cd -
cd sys/modules/vmm
make install 
cd -
cd usr.sbin/bhyve
make install 
cd -
cd usr.sbin/bhyvectl
make install 
cd -
cd usr.sbin/bhyveload
make install
 
Guys, you do understand that all these how-tos suggest you install older kernel on a more recent world, right ? Like 14/0 on a 14/1 or 15. Just to clear things out.
 
You must log in or register to reply here.
Back
Top