Solved Passthrough Nvidia GPU to Linux guest, with CUDA

[rowan194]

Hello!

I've recently discovered bhyve, and I'm now considering virtualising some of my basic "2 drives and a network connection" FreeBSD servers. That's simple enough.

More complicated: I'm wondering what the chances are of passing through a GPU (Tesla P4, headless) to a Linux guest, with its own drivers and CUDA installed, doing some pretty heavy AI/ML stuff?

Does the guest OS have full access to the passthrough GPU, seeing it as native, or is there an abstraction layer that the guest OS (and proprietary Nvidia drivers) need to support?

Asking before I spend hours battling with drivers and CUDA installs...

Thanks.

 

[rowan194]

Solution: https://dflund.se/~getz/Notes/2024/freebsd-gpu/

The page is for 14.0 and 14.1, but I was able to download the sources for 14.2 and apply the same patch (nvidia.patch) successfully. Without those patches, you'll likely have Linux complaining (repeatedly) that the card has no IRQ, and nothing will work.

Note that you still have to mask the card and specify it as passthrough to the guest. In my case:

1. pptdevs="3/0/0" in /boot/loader.conf
2. -s 11,passthru,3/0/0 as part of bhyve commandline
3. -S as part of the bhyve commandline (wire guest memory)

In addition, I've found I need to do the following to prevent the P4 from disappearing once the VM is exited (which otherwise requires a host reboot to restore the device). This is run before bhyve:

devctl clear driver -f pci0:3:0:0
devctl reset -d pci0:3:0:0
devctl detach -f pci0:3:0:0
devctl attach pci0:3:0:0

(It's possible that not all commands are needed, but after several combinations, this worked, so I stuck with it. I have this in a script.)

Hopefully this helps someone else.

$ uname -a
Linux inf1 6.5.0-25-generic #25~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Feb 20 16:09:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ nvidia-smi
Mon Mar  3 08:26:12 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P4                       Off | 00000000:00:0B.0 Off |                    0 |
| N/A   39C    P8               7W /  75W |      0MiB /  7680MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
.....