bhyve Bhyve can't pass thru any of my NVIDIA graphic cards to an Ubuntu bhyved os : the vm freezes before recognizing the disk

Zare

Well-Known Member

Reaction score: 73
Messages: 425

Thank you.

I've started working on this too several days ago.

Right now I have passthrough of GTX960 to Debian VM - default nouveau booted.
I've moved from 13-RELEASE to 13-STABLE and then implemented ROM loading in bhyve's PCI stack.


My endgame is Windows 11 so I'll keep on until job is done. Currently I'm moving debian to nvidia driver and will do graphics tests if it manages to initialize. If Linux doesn't perform stable and well it's of no use proceeding to Windows.

Edit :

Code:
root@phobos-dvm:~# dmesg | grep NVRM
[    1.906273] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  460.91.03  Fri Jul  2 06:04:10 UTC 2021
[    7.022144] NVRM: GPU 0000:00:02.0: RmInitAdapter failed! (0x12:0x45:2144)
[    7.022448] NVRM: GPU 0000:00:02.0: rm_init_adapter failed, device minor number 0
root@phobos-dvm:~# nvidia-smi
Fri Dec 10 16:06:00 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     On   | 00000000:00:02.0 Off |                  N/A |
|  0%   22C    P8     9W / 130W |      1MiB /  2001MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@phobos-dvm:~#
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

Thank you.

I've started working on this too several days ago.

Right now I have passthrough of GTX960 to Debian VM - default nouveau booted.
I've moved from 13-RELEASE to 13-STABLE and then implemented ROM loading in bhyve's PCI stack.


My endgame is Windows 11 so I'll keep on until job is done. Currently I'm moving debian to nvidia driver and will do graphics tests if it manages to initialize. If Linux doesn't perform stable and well it's of no use proceeding to Windows.

Edit :

Code:
root@phobos-dvm:~# dmesg | grep NVRM
[    1.906273] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  460.91.03  Fri Jul  2 06:04:10 UTC 2021
[    7.022144] NVRM: GPU 0000:00:02.0: RmInitAdapter failed! (0x12:0x45:2144)
[    7.022448] NVRM: GPU 0000:00:02.0: rm_init_adapter failed, device minor number 0
root@phobos-dvm:~# nvidia-smi
Fri Dec 10 16:06:00 2021  
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     On   | 00000000:00:02.0 Off |                  N/A |
|  0%   22C    P8     9W / 130W |      1MiB /  2001MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                         
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@phobos-dvm:~#

you are passing thru an old graphic card. I've read that the old nvidia graphic cards can be passed succesfully even without the latest patches. Anyway,if u get the "RmInitAdapter failed!" error,something is not good. Bhyve should be patched heavily to pass correctly the nvidia card on Windows. At the moment no one knows what's missing because Windows is based on the closed source code. And u are using an old nvidia driver. vers. 460 is too old to work. You should use an higher or equal to 465 version. Maybe you don't know what nvidia starting from 465 allows to pass thru one graphic card.
 

Zare

Well-Known Member

Reaction score: 73
Messages: 425

It definitely can't be passed through on 13-RELEASE out of the box, not considering OVMF patch.

Whether it can on 13-STABLE, I doubt, the several patches to bhyve between 13-RELEASE and 13-STABLE (at the moment) seem to be unrelated.

Only when I added the neccessary code for ROM loading and used a ROM file the GFX booted, it turned on its vents (nouveau seems to do that) and Debian proceeded to boot normally. Before the ROM patch it would start loading the OS but bhyve would fail in passthru_read().

I'll try not using the driver from apt but the latest nVidia installer - as you noticed rm_init_adapter failing is not a good sign and X11 does not manage to come up (also it's worth noting here that I haven't got monitor signal yet), with "Falied to allocate shared surface" error on X11 driver, but far down in the initialization phase. A lot of driver stuff succeeds prior to it, mainly DAC/port queries and such. I believe there's still memory mapping problems to overcome.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

For the "Falied to allocate shared surface" you should apply the MTRR patch. The nvidia driver need MTRR and PAT enabled on bhyve to work correctly. I think that you will not find this patch on the official repository of Freebsd,because it has not been accepted yet. But it is on the Corvin repository. Here :


For this patch we should thank Peter Grehan. He wrote it.
 

Zare

Well-Known Member

Reaction score: 73
Messages: 425

Thank you mr Grehan!
Yeah I've been looking in the github patches and I now have a working solution for Debian.

- Move to 13-STABLE
- BhyveX64.dsc OVMF patch for PciEnumeration
- Implement ROM loading from D33129 (autopatching stable does not work)
- Implement MTRR, bar 0 workaround and CPUID/KVM signature hack (autopatching stable does not work)
- (use nVidia-495 from the site)
 
Top