bhyve Current state of bhyve Nvidia passthrough?

syllopsium · Mar 4, 2023

From what I can see the current state is Nvidia GPU passthrough is working with patches to at least Linux and also Windows in some cases, probably with patches against 13.1. Has this changed?

Before I start messing with 13.1 with patches, 13.2, or 14.0 or looking at source code trees I thought I'd check if someone has up to date info. I have AMD passthrough working 'fine' on a dual processor Ivy Bridge system, but it's fairly useless due to the fact AMD GPUs don't like to reset after a VM exit.

Ideally looking to get FreeBSD, Linux, and Windows 10 working with passthrough. Currently using a Quadro 6000, but I have a Quadro P4000 arriving soon.

syllopsium · Mar 8, 2023

To update so far - FreeBSD 14.0 works well with the Quadro P4000 with Linux (Manjaro) and FreeBSD 14.0 CURRENT VMs. Windows next. Didn't have as much luck with the Quadro 6000.

This is possibly BIOS related. The 6000 does not have EFI support, but the P4000 does. Whether a CoreBIOS BIOS instead of a custom EDK2 EFI would change matters is an interesting question.

syllopsium · Mar 10, 2023

Further update - Windows 10 is also 'fine' provided the Nvidia drivers are installed rather than the ones supplied by Windows update. Windows Update drivers placed the GPU into a state where it wouldn't work with any VM and a devctl reset did not resolve the issue.

However, a cold reboot fixed this and the GPU is now passing between Manjaro, FreeBSD 14.0, and Windows 10 (22H2). 12GB memory allocated to the Windows VM worked fine too.

It looks like FreeBSD and Windows need vnc to be enabled to provide a VGA framebuffer in order for the Nvidia drivers to work. This is not necessary for Manjaro Linux for some reason. All three are now using the closed source Nvidia drivers, so it's not an open source vs closed source issue.

Still, this is really promising. The whole reason for this is to compare against WINE and have VM and passthrough as a fall back, so at least the fall back is now in place.

jardows · Mar 10, 2023

Good to hear that it is working on 14.0. With that being released later this year, it will FreeBSD even more compelling a platform for VM's.

dnb · Aug 21, 2023

I have read a lot of disparate topics on this forum, but did not quite understand the following. Please someone answer me briefly to simple questions:

(1) Can I passthrough NVIDIA RTX a2000 to Debian Bookworm from FreeBSD 13.2?
(2) Do I understand correctly that in FreeBSD 14.0 I can do this out of the box?

GoNeFast_01 · Aug 24, 2023

dnb said:
(1) Can I passthrough NVIDIA RTX a2000 to Debian Bookworm from FreeBSD 13.2?
(2) Do I understand correctly that in FreeBSD 14.0 I can do this out of the box?

1. In theory yes, by using = Bhyve_pci_passthru (the link says GPU not supported that's why you need to test or do patches) you will need to test with RTX a2000 and 13.2.
2. What OP stated its that for him FBSD 14.0 + Passthrough with his NVIDIA Quadro P4000 is working, you will have to test with RTX a2000 (hopefully it works out the box like your question implies).

(My journey) I will be testing with some old Nvidia cards (1080ti, 2080ti) but it seems functional now in 13.2/13.1 and in FreeBSD 14.0 should increase the support (if patches are added).

dnb · Oct 17, 2023

GoNeFast_01 said:
1. In theory yes, by using = Bhyve_pci_passthru (the link says GPU not supported that's why you need to test or do patches) you will need to test with RTX a2000 and 13.2.
2. What OP stated its that for him FBSD 14.0 + Passthrough with his NVIDIA Quadro P4000 is working, you will have to test with RTX a2000 (hopefully it works out the box like your question implies).

(My journey) I will be testing with some old Nvidia cards (1080ti, 2080ti) but it seems functional now in 13.2/13.1 and in FreeBSD 14.0 should increase the support (if patches are added).

Here is a link to a recent presentation at the EuroBSDcon 2023 conference. The link points precisely to the point about support for video cards from various manufacturers. And the speaker states that NVIDIA is not supported:

View: https://youtu.be/eurBCPj65oI?t=439

As far as I understand (and correct me if this is not so), this speaker is practically the only one currently working on this problem. How did NVIDIA cards still work?

I'd like to know more details, but it's not clear where to look. Details I'm talking about:

1 - specific versions of FreeBSD code
2 - specific version of Linux distribution

Maybe someone here who was able to get NVIDIA to work will give a report that will allow me to reproduce the work.

aragats · Dec 5, 2023

What am I doing wrong when trying to passthru RTX A2000 in FreeBSD 14.0?

Code:

% pciconf -lv | grep -A3 ppt
ppt0@pci0:33:0:0:       class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2531 subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 [RTX A2000]'
    class      = display

The bhyve launching script:

Code:

....
GPU="33/0/0"
....
-s 11,passthru,$GPU
....

and bhyve failes with:

Code:

bhyve: PCI device at 33/0/0 is not using the ppt(4) driver
device emulation initialization error: Device busy

Thanks for ideas!

driesm · Dec 5, 2023

You need to mask the GPU with the ppts driver in /boot/loader.conf, this way the nvidia graphics card does not attach to the host.

vmm(4)

man.freebsd.org

aragats · Dec 5, 2023

driesm said:
You need to mask the GPU with the ppts driver in /boot/loader.conf, this way the nvidia graphics card does not attach to the host.

Well, the fact that pciconf's output about shows ppt is an evidence of that I already have it in /boot/loader.conf:

Code:

% grep ppt /boot/loader.conf
pptdevs="33/0/0"

yuripv79 · Dec 5, 2023

aragats said:
and bhyve failes with:

Anything in dmesg? Does vmm module load/attach successfully?

aragats · Dec 5, 2023

Nothing in dmesg, vmm works fine: this host runs two other VMs without issues.

aragats · Dec 5, 2023

No tot sure whether the following can shed some light, and how is this related to pciconf's output of "ppt0@pci0:33:0:0" in terms of numbers?

Code:

 % dmesg | grep -B2 -A4 ppt
pcib4: <ACPI PCI-PCI bridge> at device 3.1 on pci3
pci4: <ACPI PCI bus> on pcib4
ppt0 port 0x6000-0x607f mem 0xe0000000-0xe0ffffff,0xc0000000-0xcfffffff,0xd0000000-0xd1ffffff irq 86 at device 0.0 on pci4
hdac0: <NVIDIA (0x228e) HDA Controller> mem 0xe1080000-0xe1083fff irq 87 at device 0.1 on pci4
pcib5: <ACPI PCI-PCI bridge> at device 7.1 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 8.1 on pci3

yuripv79 · Dec 5, 2023

You probably need to passthru that sound device as well, which is usually on function 1 on nvidia gpus. ESXi automatically selects it when trying to passthru the GPU itself.

aragats · Dec 5, 2023

yuripv79 said:
You probably need to passthru that sound device as well...

That's a great idea, will try tonight when folks will stop using those VMs.

aragats · Dec 6, 2023

Passing through both Nvidia devices doesn't help, I still get "busy" on the main one:

Code:

bhyve: PCI device at 33/0/0 is not using the ppt(4) driver

I wonder if it's related to the fact that Nvidia is the only video adapter installed: the CPU is AMD Ryzen Threadripper 3960X and doesn't have integrated graphics.

Phishfry · Dec 6, 2023

Are you starting your bhyve instances with bhyveload or EDK2 UEFI firmware?

aragats · Dec 6, 2023

Phishfry said:
Are you starting your bhyve instances with bhyveload or EDK2 UEFI firmware?

I'm using UEFI firmware: /usr/local/share/uefi-firmware/BHYVE_UEFI.fd.

yuripv79 · Dec 6, 2023

aragats said:
I wonder if it's related to the fact that Nvidia is the only video adapter installed

Could be, I wonder if the console is attached to it anyway despite the gpu being ppt'ed. May be try forcing serial or null console (if you don't need boot output and/or system console)?

loader.conf:

Code:

console="nullconsole"

aragats · Dec 6, 2023

yuripv79 said:
Code:

console="nullconsole"

Doesn't help. I wonder why I see 8 getty running with nullconsole though:

Code:

% ps aux | grep getty
root    2115    0.0  0.0  12832   2372 v0  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv0
root    2116    0.0  0.0  12832   2368 v1  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv1
root    2117    0.0  0.0  12832   2368 v2  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv2
root    2118    0.0  0.0  12832   2372 v3  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv3
root    2119    0.0  0.0  12832   2372 v4  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv4
root    2120    0.0  0.0  12832   2372 v5  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv5
root    2121    0.0  0.0  12832   2368 v6  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv6
root    2122    0.0  0.0  12832   2368 v7  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv7

nunotex · Dec 16, 2023

A few months ago I did tried nvidia passthru on a laptop with intel integrated and dedicated nvidia.
I remember that to use ppt on nvidia I must boot OS with intel selected (leaving nvidia unnused).

At that time I passthru nvidia to win10 and it was recognized but installation of nvidia drivers failed at some point.

Any tips or aditional information about:

 vgapci0@pci0:1:0:0:     class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1f99 subvendor=0x17aa subdevice=0x3fa5

    vendor     = 'NVIDIA Corporation'

    device     = 'TU117M [GeForce GTX 1650 Mobile / Max-Q]'

    class      = display

    subclass   = VGA

?

nunotex · Dec 16, 2023

 /boot/loader.conf.local:

vmm_load="YES"

pptdevs="1/0/0 1/0/1"



ppt0@pci0:1:0:0:        class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1f99 subvendor=0x17aa subdevice=0x3fa5

    vendor     = 'NVIDIA Corporation'

    device     = 'TU117M [GeForce GTX 1650 Mobile / Max-Q]'

    class      = display

    subclass   = VGA

ppt1@pci0:1:0:1:        class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10fa subvendor=0x17aa subdevice=0x3ffb

    vendor     = 'NVIDIA Corporation'

    class      = multimedia

    subclass   = HDA



bhyve:

-s 6:0,passthru,1/0/0 -s 6:1,passthru,1/0/1

- Installed nvidia drivers without problems
- graphic card is detected but is deactivated with a triangle and a "!":
"Windows stopped this device... code 43"
- hda is detected and shows no problem

hoewweken · Dec 19, 2023

Hello nunotex!
it was mentioned that it may work better if you enable VNC for your vm, have you tried that?

nunotex · Dec 20, 2023

hoewweken said:
Hello nunotex!
it was mentioned that it may work better if you enable VNC for your vm, have you tried that?

vnc was on by default as I was running win11 with tigervnc.

aragats · Dec 20, 2023

I installed another graphics card in the same box. I'm not sure whether there is a setting which one is the "default", but the new one does display the console at boot.
However, I'm still getting "device is busy" error when trying to passthru the original one.
Another question: the one I want to passthru is pci0:33:0:0 and the new one pci0:1:0:0. Why those numbers are so different? I'm not really familiar with PCIe, but it seems strange to me that the devices from the same vendor have so different IDs:

Code:

 % pciconf -lv | grep -A2 vga
vgapci0@pci0:1:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x06fd subvendor=0x10de subdevice=0x062e
    vendor     = 'NVIDIA Corporation'
    device     = 'G98 [Quadro NVS 295]'

% pciconf -lv | grep -A2 ppt
ppt0@pci0:33:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2531 subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 [RTX A2000]'
--
ppt1@pci0:33:0:1:    class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x228e subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 High Definition Audio Controller'