Bhyve GPU pass-through

ludovitkramar

New Member

Reaction score: 4
Messages: 3

Hello! I've been trying to get my RX570 working on an Asrock H97 Pro 4 motherboard, I've passed the GPU successfully (with vm-bhyve) to an archlinux vm, but it seems like the amdgpu is not working correctly, I can only get stable graphics with acceleration disabled, and everything is extremely slugish.

In dmesg I get the following messages: (partial)

Code:
[  473.038005] [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
[  475.444739] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=17, emitted seq=20
[  475.444981] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process WebKitWebProces pid 1568 thread WebKitWebP:cs0 pid 1581
[  475.445197] amdgpu 0000:00:06.0: amdgpu: GPU reset begin!
[  476.042614] amdgpu: cp is busy, skip halt cp
[  476.227217] amdgpu: rlc is busy, skip halt rlc
[  476.228234] amdgpu 0000:00:06.0: amdgpu: BACO reset
[  476.845294] amdgpu 0000:00:06.0: amdgpu: GPU reset succeeded, trying to resume
[  476.845807] [drm] PCIE GART of 256M enabled (table at 0x000000F400000000).
[  476.845829] [drm] VRAM is lost due to GPU reset!
[  477.072011] [drm] UVD and UVD ENC initialized successfully.
[  477.174992] [drm] VCE initialized successfully.
[  477.179435] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow start
[  477.179451] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow done
[  477.179453] [drm] Skip scheduling IBs!

it doesn't matter what I do, after a while the driver crashes and resets itself, meanwhile the screen is a glitched color noise mess, it is impossible to make out anything. That is the case on xorg and on wayland.

Is it more likely for this to be a bhyve issue or a driver inside the vm issue?
 

ziomario

Daemon

Reaction score: 138
Messages: 1,099

SirDice : is not actually totally true. There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.
 

ziomario

Daemon

Reaction score: 138
Messages: 1,099

Hello! I've been trying to get my RX570 working on an Asrock H97 Pro 4 motherboard, I've passed the GPU successfully (with vm-bhyve) to an archlinux vm, but it seems like the amdgpu is not working correctly, I can only get stable graphics with acceleration disabled, and everything is extremely slugish.

In dmesg I get the following messages: (partial)

Code:
[  473.038005] [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed to initialize parser -125!
[  475.444739] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, signaled seq=17, emitted seq=20
[  475.444981] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process WebKitWebProces pid 1568 thread WebKitWebP:cs0 pid 1581
[  475.445197] amdgpu 0000:00:06.0: amdgpu: GPU reset begin!
[  476.042614] amdgpu: cp is busy, skip halt cp
[  476.227217] amdgpu: rlc is busy, skip halt rlc
[  476.228234] amdgpu 0000:00:06.0: amdgpu: BACO reset
[  476.845294] amdgpu 0000:00:06.0: amdgpu: GPU reset succeeded, trying to resume
[  476.845807] [drm] PCIE GART of 256M enabled (table at 0x000000F400000000).
[  476.845829] [drm] VRAM is lost due to GPU reset!
[  477.072011] [drm] UVD and UVD ENC initialized successfully.
[  477.174992] [drm] VCE initialized successfully.
[  477.179435] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow start
[  477.179451] amdgpu 0000:00:06.0: amdgpu: recover vram bo from shadow done
[  477.179453] [drm] Skip scheduling IBs!

it doesn't matter what I do, after a while the driver crashes and resets itself, meanwhile the screen is a glitched color noise mess, it is impossible to make out anything. That is the case on xorg and on wayland.

Is it more likely for this to be a bhyve issue or a driver inside the vm issue?

I suggest to report this behavior to : freebsd-virtualization@freebsd.org
 
OP
L

ludovitkramar

New Member

Reaction score: 4
Messages: 3

freezr

Well-Known Member

Reaction score: 146
Messages: 340

Hi guys, do you know if the GPU-passthrough would work also with an eGPU connected through a Thunderbolt port?

Thanks,
TGL
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 14,036
Messages: 40,763

Would it take a long time for these patches to get into the binary packages?
Merged into existing releases is unlikely. These things are tested in -CURRENT first. Then they may or may not be MFC'ed to -STABLE. If they are MFC'ed to -STABLE they'll be included in the next minor release of that major branch.
 

hsnyder

New Member

Reaction score: 9
Messages: 10

SirDice : is not actually totally true. There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.
Do you have a link to the patch that you used? I had a look in Phabricator but I only saw the GVT-d patches and the AMD patch that was linked by ludovitkramar...

 

Watitsthis

Member

Reaction score: 17
Messages: 27

Looking for patches I've found this patch that's for review, I've never applied one but I'm sure I could figure it out somehow.
https://reviews.freebsd.org/D27456

Would it take a long time for these patches to get into the binary packages?
Appreciate this very much.. Thank you

There are a lot of good and working patches that have been proposed and they worked great with the amd and the nvidia graphic cards. A lot of them haven't been accepted by the freebsd developers,but they work. I'm passing thru my nvidia rtx 2080 ti with the nvidia patches and it is working great. For sure,it's under my own responsability if something will be wrong with my PC. But I took the risk and everything is going ok. So,if you want to pass thru your amd gpu,you can do that. It works, just right now. But you should know how to apply them.

I am going to do some more testing with these patches I have a couple GPU (Nvidia mostly) laying around I could test emulation .... Were you able to play STABLE games or high performing programs on your nvidia passthru ? ziomario
 

ziomario

Daemon

Reaction score: 138
Messages: 1,099

Appreciate this very much.. Thank you



I am going to do some more testing with these patches I have a couple GPU (Nvidia mostly) laying around I could test emulation .... Were you able to play STABLE games or high performing programs on your nvidia passthru ? ziomario

yes,but only for Linux. Those patches aren't mine. I'm only a tester. They come from the mind of a really nice developer,Corvin.
 

hsnyder

New Member

Reaction score: 9
Messages: 10

Hi everyone. I was able to get GPU passthrough working with an nvidia GPU passed through to a Linux guest (ubuntu 20.04). I used the patches posted above, authored by Corvin Köhne of Beckhoff, but I didn't apply them manually. Instead, starting with a relatively clean install of 13.0-RELEASE, I cloned the beckhoff FreeBSD source code mirror at https://github.com/Beckhoff/freebsd-src, specifically Corvin's branch phab/corvink/13.0-nvidia-wip. Note that there's another branch that's just called 13.0-nvidia, without the '-wip', which did not work for me (though it's possible that's my own fault).

Anyways, with that repo, I rebuilt world & kernel, and installed (I literally just followed the steps in the rebuild world part of the freebsd handbook, I've never done it before and had no special knowledge). Then, I followed the instructions on this page https://wiki.freebsd.org/bhyve/pci_passthru to pass the device through to the VM. If you're on an AMD system like me, be sure to check the "Additional notes" at the bottom of the page.

I also had to use Corvin's modified UEFI bootrom. I didn't build it myself, he just sent me the binary, which is too big to attach to this post (~3MB) otherwise I would. If I understand correctly, it can be produced from the code here: https://github.com/Beckhoff/edk2/commits/phab/corvink/gvt-d
Anyways, the modified UEFI filename was BHYVE_BHF_CODE.fd in my case, so I had the following in my bhyve command:
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd

Note also that Corvin said that the current patch won't work for a windows guest yet, only FreeBSD or Linux. But if I understand correctly, he's still working on Windows support.
 
Last edited:

ziomario

Daemon

Reaction score: 138
Messages: 1,099

Yes,we are working on the passthrough for Windows for the Nvidia gpus (the AMD gpus already work),but we are on the high seas. I think that 13.0-nvidia-wip is the code work in progress for the passthrough for Windows....

----> So it looks like the GPU is limited to PCIe gen 1 speed ?

I don't know,but I'm interested to know what's the answer.
 
Last edited:
Top