bhyve Current state of bhyve Nvidia passthrough?

I have two nvidia cards, 1080ti and tesla p100. They worked somehow on 13.2, I was able to passthru these cards to VM with ubuntu and nvidia drivers. In my case the system with 1080ti and p100 on 13.2 is not stable, sometimes happens that whole system freez, or cpu soft lockup occures for infinite time inside VM but it works.

After upgrading to 14.0 (using exactly the same VM, the same drivers, the same bhyve configuration) on VM dmesg shows:
[ 77.208984] NVRM: Can't find an IRQ for your NVIDIA card!
[ 77.212697] NVRM: Please check your BIOS settings.
[ 77.212699] NVRM: [Plug & Play OS] should be set to NO
[ 77.212700] NVRM: [Assign IRQ to VGA] should be set to YES
[ 77.212702] nvidia: probe of 0000:00:07.0 failed with error -1
[ 77.212733] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 77.212734] NVRM: None of the NVIDIA devices were initialized.
[ 77.213269] nvidia-nvlink: Unregistered Nvlink Core, major device number 237
Click to expand...
and repeating system messages:
[ 206.437547] NVRM: Can't find an IRQ for your NVIDIA card!
[ 207.439361] NVRM: Can't find an IRQ for your NVIDIA card!
[ 208.455424] NVRM: Can't find an IRQ for your NVIDIA card!
[ 209.451508] NVRM: Can't find an IRQ for your NVIDIA card!
[ 210.464152] NVRM: Can't find an IRQ for your NVIDIA card!
[ 211.475571] NVRM: Can't find an IRQ for your NVIDIA card!
Click to expand...
I can reproduce this on two different computers (both based on Intel) and two different Nvidia cards (1080ti, p100).
I am looking for any hint.
 
I've been successfully utilizing Corvin's Nvidia passthrough code since version 13.0, specifically with my Quadro K2200 card and Linux (Debian) as the guest OS. Remarkably, I haven't encountered any issues yet. Detailed steps were previously shared by someone in the forum. Essentially, the process involves fetching the codes from https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip for FreeBSD 14.0 compatibility. If targeting version 13.1, simply locate the 13.1/nvidia-wip branch. Then, compile the source code following the instructions outlined in the FreeBSD manual. Finally, proceed with standard passthrough steps as outlined in the bhyve manual. Following these steps should result in a functional Nvidia GPU setup!
 
The code for 14.0 with Corvin patches seems a little bit broken. I needed to do some minor fixes to compile it:
Diff: 
diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index 379b1255a79f..f74adfa2f84b 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -1171,7 +1171,7 @@ passthru_write(struct pci_devinst *pi, int baridx, uint64_t offset, int size,
        } else if (baridx == 0 &&
            pci_get_cfgdata16(pi, PCIR_VENDOR) == PCI_VENDOR_NVIDIA &&
            pci_get_cfgdata8(pi, PCIR_CLASS) == PCIC_DISPLAY) {
-               passthru_cfgwrite(ctx, vcpu, pi, offset - 0x88000, size, value);
+               passthru_cfgwrite(pi, offset - 0x88000, size, value);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1206,7 +1206,7 @@ passthru_read(struct pci_devinst *pi, int baridx, uint64_t offset, int size)
                /* dummy read to MMIO because hw might depend on it */
                memcpy(&val, nvidia_bar0 + offset, size);
 
-               passthru_cfgread(ctx, vcpu, pi, offset - 0x88000, size, (uint32_t *)&val);
+               passthru_cfgread(pi, offset - 0x88000, size, (uint32_t *)&val);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1299,14 +1299,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
@@ -1324,14 +1324,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
(END)
I installed the new kernel and bhyve (only this one file, not the whole world). Now nvidia 1080ti backed to life on 14.0. My guest is ubuntu 22.04 with driver 535.129.03.
I really hope it will be more stable than on 13.2 (without patches) but it needs to do more long term tests.
 
Thank you to everyone involved in developing and testing this work. It is very much needed and appreciated.
My question is: does anyone have a single patch that can be applied to 14.0/STABLE?
I've been a bit disconnected therefore am wondering why (if something of this magnitude exists) it never made it into FreeBSD 14? Thank you.
 
As far as I know there were more patches, and most of them have been applied to 14.0.
For now there are two patches indeed. One is for bhyve (it refers to "dummy bar 0" issue) and the second one is a kernel patch that changes hypervisor signature from bhyve to kvm signature.

I do not know much about "bar 0" and gpu, but for obvious reason you cannot change hypervisor signature permanently. There is a suspicious that nvidia drivers can work differently for different hypervisor.

I tested 1080ti a little more, and it seems to work stable. There is only a problem after restarting VM. That means if I want to restart VM with gpu attached to it, I need to restart the hole computer otherwise the gpu not work.
 

Attachments

  • 14.0-stable-nvidia.patch.txt
    6.3 KB · Views: 102
Thanks for the patch - but you can easily see why it didn't make it into 14.0. With it applied although Linux passthrough now works, attempting another VM start afterwards almost always results in a hard lock of the entire system at least here (Ivy Bridge, C602 chipset).

The hypervisor signature change isn't required, only the modifications in the bhyve directory. I don't remember it being this unstable in some pre 14.0 builds. It's also currently only Linux that's affected, without the patch Windows and FreeBSD VMs are quite happy to switch between each other, at least in the case of an Nvidia CPU with proprietary (Windows) and open source (FreeBSD) drivers on my system.

Unfortunately Linux passthrough is what I need, because I'm trying to compare FreeBSD WINE to Linux WINE, as I suspect Linux WINE is more compatible than FreeBSD (even for the same version).
 
You must log in or register to reply here.
Back
Top